5 ms·

I'd be interested to know how they came to realize what was missing. Did they read the Jenkins docs more thoroughly? Post on a mailing list? See something on

I'd be interested to know how they came to realize what was missing. Did they read the Jenkins docs more thoroughly? Post on a mailing list? See something on StackOverflow? Hire a consultant?

pushrax 6y ago
I would expect that internally someone profiled the build (i.e. looked at timestamps) and then either profiled git, or just looked at the logs and did some guessing/research. This didn't seem like it would be complicated to find once you realize the time is spent in git. Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there are a lot of stale undeleted branches.
- marta_morena_28 6y ago
  It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanation I can come up with is that the whole build takes hours anyway, otherwise there is no way you wouldn't notice this sooner.
  - oefrha 6y ago
    Yep, I’m shocked that it had to be bloated to 40min before they even thought about fixing it. Anyone who has used Jenkins for nontrivial builds must have had the experience of staring at the slowly expanding session log screen? It doesn’t take any “profiling” to realize git clone’s taking forever.
  - pushrax 6y ago
    It takes >12h to build Windows on the MS build platform. On a single core, Chromium surely takes hours to build. Though I agree that 40min for the repository in question is highly suspect.
    - masklinn 6y ago
      > On a single core, Chromium surely takes hours to build. Earlier this year Bruce Dawson had a post indicating that it took about a CPU-day, though coalescing files (“jumbo builds”) significantly reduces build time (we’re talking down to 5h), however that’s at the expense of incremental building, and it constrains the code as you can get symbol collisions between the coalesced files.
    - Dylan16807 6y ago
      I read that comment as only applying the "single core" part to the Linux kernel, not Chromium.
  - detaro 6y ago
    Remember "build" time likely also includes test runs, packaging for deployment, ... 40 mins is easy to get to, and has nothing to do with "on a single core".
    - sumedh 6y ago
      You can still see which stages are taking a long time.
  - maccard 6y ago
    I work in games, a clean sync of our project takes well over an hour (and if you're at home it takes multiple hours), and compiling takes a large amount of time. We use lots of tricks - "unity" builds with out build system detecting modified files and compiling them standalone as an example. On my last workstation (2x intel xeon gold) it took about 30 minutes to compile
  - jeromenerf 6y ago
    I wonder if people went into the habit of synchronizing git pushes with socializing breaks, with the proverbial excuse of "yeah, compiling ...". On day, someone forgot to brief the foreign intern about the necessity of breaks, intern fixes the issue, pointy-hair-boss gets wind of the news, old crew gets fired, new intern gets promoted and fixes also the Pinterest spam on google images.
    - znpy 6y ago
      > new intern gets promoted and fixes also the Pinterest spam on google images A man can dream.
  - ImprovedSilence 6y ago
    I remember hearing facebook was on the order of hours (6+) to build.
jeromenerf 6y ago
I would bet on "the new recruit found it, 45 minutes in in the onboarding process".
- conradludgate 6y ago
  My first task at my current job, after familiarising myself with the codebase was to improve the CI pipeline
- Aloha 6y ago
  I think everyone has done this, sometimes it really does take a second set of eyes
  - ric2b 6y ago
    In my experience it only takes someone who is not drowning in a backlog of tasks yet. That tends to be the beginners during the onboarding weeks.
- koliber 6y ago
  The beauty and power of the beginner's mind. It can see things that were there all along, but everyone who has been there has developed a blindness to.
bluedino 6y ago
I see this kind of stuff at companies where only one or two developers work on a project, or the team working on it hasn't had much experience on other projects. An example would be a company I worked for who ran a pretty standard LAMP setup but had never heard of memcached. Simply adding that reduced the database load by like 90%.
aeyes 6y ago
The same way it always happens to me: Over time build time creeps up and you don't really look at this code thinking "I implemented shallow clone years ago so there is nothing I can do, it's slower because we have more code." Until you or some other person looks what the code is doing. It could also be that it was a new hire. I shallow clone a huge monorepo similar in commits/branches and it takes seconds. My experience would instantly tell me that something is worth looking into.

I'd be interested to know how they came to realize what was missing. Did they read the Jenkins docs more thoroughly? Post on a mailing list? See something on StackOverflow? Hire a consultant?

pushrax 6y ago

I would expect that internally someone profiled the build (i.e. looked at timestamps) and then either profiled git, or just looked at the logs and did some guessing/research. This didn't seem like it would be complicated to find once you realize the time is spent in git. Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there are a lot of stale undeleted branches.

marta_morena_28 6y ago
It really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanation I can come up with is that the whole build takes hours anyway, otherwise there is no way you wouldn't notice this sooner.
- oefrha 6y ago
  Yep, I’m shocked that it had to be bloated to 40min before they even thought about fixing it. Anyone who has used Jenkins for nontrivial builds must have had the experience of staring at the slowly expanding session log screen? It doesn’t take any “profiling” to realize git clone’s taking forever.
- pushrax 6y ago
  It takes >12h to build Windows on the MS build platform. On a single core, Chromium surely takes hours to build. Though I agree that 40min for the repository in question is highly suspect.
  - masklinn 6y ago
    > On a single core, Chromium surely takes hours to build. Earlier this year Bruce Dawson had a post indicating that it took about a CPU-day, though coalescing files (“jumbo builds”) significantly reduces build time (we’re talking down to 5h), however that’s at the expense of incremental building, and it constrains the code as you can get symbol collisions between the coalesced files.
  - Dylan16807 6y ago
    I read that comment as only applying the "single core" part to the Linux kernel, not Chromium.
- detaro 6y ago
  Remember "build" time likely also includes test runs, packaging for deployment, ... 40 mins is easy to get to, and has nothing to do with "on a single core".
  - sumedh 6y ago
    You can still see which stages are taking a long time.
- maccard 6y ago
  I work in games, a clean sync of our project takes well over an hour (and if you're at home it takes multiple hours), and compiling takes a large amount of time. We use lots of tricks - "unity" builds with out build system detecting modified files and compiling them standalone as an example. On my last workstation (2x intel xeon gold) it took about 30 minutes to compile
- jeromenerf 6y ago
  I wonder if people went into the habit of synchronizing git pushes with socializing breaks, with the proverbial excuse of "yeah, compiling ...". On day, someone forgot to brief the foreign intern about the necessity of breaks, intern fixes the issue, pointy-hair-boss gets wind of the news, old crew gets fired, new intern gets promoted and fixes also the Pinterest spam on google images.
  - znpy 6y ago
    > new intern gets promoted and fixes also the Pinterest spam on google images A man can dream.
- ImprovedSilence 6y ago
  I remember hearing facebook was on the order of hours (6+) to build.

jeromenerf 6y ago

I would bet on "the new recruit found it, 45 minutes in in the onboarding process".

conradludgate 6y ago
My first task at my current job, after familiarising myself with the codebase was to improve the CI pipeline
Aloha 6y ago
I think everyone has done this, sometimes it really does take a second set of eyes
- ric2b 6y ago
  In my experience it only takes someone who is not drowning in a backlog of tasks yet. That tends to be the beginners during the onboarding weeks.
koliber 6y ago
The beauty and power of the beginner's mind. It can see things that were there all along, but everyone who has been there has developed a blindness to.

bluedino 6y ago

I see this kind of stuff at companies where only one or two developers work on a project, or the team working on it hasn't had much experience on other projects. An example would be a company I worked for who ran a pretty standard LAMP setup but had never heard of memcached. Simply adding that reduced the database load by like 90%.

aeyes 6y ago

The same way it always happens to me: Over time build time creeps up and you don't really look at this code thinking "I implemented shallow clone years ago so there is nothing I can do, it's slower because we have more code." Until you or some other person looks what the code is doing. It could also be that it was a new hire. I shallow clone a huge monorepo similar in commits/branches and it takes seconds. My experience would instantly tell me that something is worth looking into.