18 ms·
A one-line change decreased our build times by 99%
- villgax 6y ago60K pulls per day for 100 commit in a day? What tests are being done that can't leverage earlier pulls?
- JOnAgain 6y agoI think it takes some real humility to post this. No doubt someone will follow up with an “of course...” or “if you don’t understand the tech you use...” comment. But thank you for this. It takes a bit of courage to point out you’ve been doing something grotesquely inefficient for years and years.
- hashkb 6y agoThey are a publicly traded company. They have a team dedicated to engineering support. A better article would include a management and hiring postmortem. It's shocking, really. Humility is nice, but competency is also nice.
- ponker 6y agoThey did a billion dollars of revenue last year, their management and hiring systems seem to be getting the job done.
- sk5t 6y agoYep--although multiple unpleasant experiences with pinterest have spurred me to permaban it from search engine results and smite it with network filters, somewhat wasteful CI/CD pipelines have clearly not prevented the company from flourishing.
- Aeolun 6y agoI don’t know, I consider myself fairly competent but I’d never even considered that. It’s just not so relevant until your repo is multiple gigabytes big. Still, I’ll see if it works for our pipelines, and we can get our clone from 20s to 1s
- hashkb 6y agoGood teams profile everything. This team's only goal is to support other engineers. Build time is a huge issue for every ops team. Missing this for so long is wasted money that's easy to calculate. We can be nice to people while still having high standards. It's a missed opportunity for a deeper postmortem, and it's bland content at best.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- jacquesm 6y agoI think you live in a highly theoretical parallel universe. The one I occupy is the one where 'good teams' profile those things that take too long. Take yourself as an example: in spite of the wide availability of free certificates you are still hosting your domain without using a secure transport layer. Some would take that as incompetence. Others would assume you have more stuff on your plate rather than that you don't have high standards.
- sdoering 6y agoAnd others would assume, that there are other philosophies out there regarding ssl everywhere. So the question is, who's POV has more validity and logical rigor attached to it. I actually can't see any side winning here on a purely logical level. Only on an ideological level. At least as long as we are talking about consuming public information. Am I in favor of the aggressiveness of OP in other posts? No. Am I using SSL myself. Hell yes. Nonetheless, I understand that there are people who feel that consuming public information like on a private homepage is nothing that necessitates using SSL. Even if I myself have a different ideology/value set governing my decision. I once heard the comparison that it is like the difference of sending a letter and sending a picture postcard. Not sure if I buy into that, but I can't argue against it on a purely rational basis.
- argc 6y agoThis is neither incompetence nor surprising. Maybe you’ve only worked at large companies who have had time to optimize things for years (and even then, I see grotesque software decisions at my large company quite often). Try accepting that software is often written poorly optimized on the first pass, for good reason, and learn to celebrate the wins without needing to shame someone.
- hashkb 6y agoThis is Pinterest. Every org I've worked at has been smaller. There's space between shame and ignoring mistakes. The purpose of this post is not to educate. There's nothing in here that anyone can use to improve. It's just marketing.
- phist_mcgee 6y agoWell I learned something from this article, and I thought I had a good handle on CI/CD, so either I am incredibly stupid and shouldn't be reading these 'nothing' articles, or maybe there is so much to learn it's impossible to know it all.
- sytringy05 6y agoIf you dont have massive repos, then this is the sort of thing that not often a big problem. Also - if you are using things like gitlab runners, you might be in the same AZ and even large repos clones are fast. And it is impossible to know it all, I like these articles just for the differing ways people work
- phist_mcgee 6y agoIt sounds like GP has a bit of a chip on their shoulder when it comes to feeling like you're not worthy if you don't know everything.
- stjohnswarts 6y agoAre you just an angry ex Pinterest employee? There is something to learn here and a reminder to pay more attention even when you're knee deep in other tasks. Besides the obvious feature/limitation of git that the author points out.
- swsieber 6y agoPeople praise my git skills at work (among other things) when they come to me for git help. My response is always the same: I've just run into these bugs more often than they.
- fizixer 6y agoFully agree, and the mindless HN downvote sheep bandwagon is in full effect. Working at Pinterest and acting humbled by learning basic stuff on the job? I play the world's smallest violin for how hard and stressful your work is. Poor babies. Somebody give this engineering team a participation trophy.
- jedberg 6y agoWhy are you so angry about this? You've commented throughout this post about how this is boring and the Pinterest team is incompetent. Why? I found it quite interesting. I've been working in deployments for over 20 years at some pretty big places, and never really though about this before. I now have a new tool in my toolbox, and I'm quite happy about it.
- hashkb 6y agoIn general I'm frustrated that rigor, standards, etc are out the window in favor of all this warm fuzziness. I guess I might be angry... The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. Time, money, electricity, customers. We lose good engineers and tell ourselves they were a bad fit. We push crap on users just to sell ads. Then we write engineering posts to brag about fixing our own mistakes. It's a terrible shame and I speak up about it to remind everyone there was a time when RTFM would be the only response to this. Edit: rate limited but one last thing: are we this forgiving of Equifax when they oopsie our data? Seeing this would immediately make me wonder if anything I have shared with Pinterest is safe. That's why they owe us a postmortem and not a thirst trap.
- pavel_lishin 6y ago> The culture change in our industry, towards warm fuzzies and away from tech screens, results in calculable waste. I have not noticed, in the past decade, any move away from tech screens whatsoever.
- uglycoyote 6y agoAre we talking about technological screens, like LCD screens here?
- xdavidliu 6y agoI believe they mean screens in the sense of technical interviews to assess candidates' technical ability.
- kevinventullo 6y agoA team discovers a major efficiency win requiring minimal engineering effort and your response is to... punish them?
- Hnrobert42 6y agoShocking? Incompetent? A hiring postmortem? Really? AYFKM?
- projektfu 6y agoI'd be interested to know how they came to realize what was missing. Did they read the Jenkins docs more thoroughly? Post on a mailing list? See something on StackOverflow? Hire a consultant?
- pushrax 6y agoI would expect that internally someone profiled the build (i.e. looked at timestamps) and then either profiled git, or just looked at the logs and did some guessing/research. This didn't seem like it would be complicated to find once you realize the time is spent in git. Also, this probably has been an exponentially increasing problem, and wasn't really a priority to solve until relatively recently. I would bet there are a lot of stale undeleted branches.
- marta_morena_28 6y agoIt really doesn't sound complicated to find, unless you just have a handsoff approach to building things and just don't care as long as "something" comes out on the other end. What makes me wonder however is this: 40 min made them look into this? I mean 40 min is crazy long. What builds this long? Chrome, Windows, Linux Kernel on a single core? This should have been raising red flags much earlier. The only explanation I can come up with is that the whole build takes hours anyway, otherwise there is no way you wouldn't notice this sooner.
- oefrha 6y agoYep, I’m shocked that it had to be bloated to 40min before they even thought about fixing it. Anyone who has used Jenkins for nontrivial builds must have had the experience of staring at the slowly expanding session log screen? It doesn’t take any “profiling” to realize git clone’s taking forever.
- pushrax 6y agoIt takes >12h to build Windows on the MS build platform. On a single core, Chromium surely takes hours to build. Though I agree that 40min for the repository in question is highly suspect.
- bald42 6y agoI don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.
- mschuster91 6y agoEphemeral CI runners. I have the same problem at work - 4GB repository that is redownloaded on every single pipeline run. Another reason (which is why we went for ephemeral runners in the first place...) is that if you have stuff that mounts a directory from the repository directory as a volume in a Docker container (e.g. for processing data), you may end up with the Docker container frying permissions in the repo folder (e.g. 0:0 owned files). Now, you can put a cleanup step as part of the CI (=docker run --rm -v $(pwd):/mnt sh -c 'chown -R $runner_uid:$runner_gid)... but unfortunately, Gitlab does not allow a "finally" step that always gets run, so in case the processing fails, the build gets aborted, the server hosting the runner crashes, ... anything happens, the permissions will be fried, and a sysadmin will need to manually intervene. An ephemeral runner using docker:dind however? It simply gets removed.
- ForHackernews 6y agoI don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI. It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.
- DougBTX 6y agoIt is interesting that we trust our tools so little. A git hash is a pretty robust way to know whether the code in the repo is what it is supposed to be, so a "git fetch" rather than a fresh "git clone" should be safe, but we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step is to delete the node_modules folder, as otherwise it can't be trusted to update correctly.
- TylerE 6y ago@Dang, can we get an edit? This did NOT slash build times 99%, but rather time to do a git pull.
- lemax 6y agoIf build includes a git pull, maybe it did.
- hashkb 6y agoNitpick... if 99% of your build time is consumed by preparing the workspace, that's the story. This isn't interesting to anyone who doesn't have that exact problem. Most people who click this won't find it interesting.
- deleted 6y ago[deleted]
- freedomben 6y agoFrom the article: > We found that setting the refspec option during git fetch reduced our build times by 99%. Seems pretty clear to me that build times were reduced by 99% as a result of cutting the git fetch times significantly (but exacyt number is not give). The headline looks correct to me.
- TylerE 6y agoFTA: "This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result" Unless their build is 100% git pull time, this did not reduce build time by 99%.
- xattt 6y agoWill this mean even more Google image search spam‽
- jtchang 6y agoThis is good info. Need to check my own build pipelines now and see if we are just blindingly cloning everything or not. 40 minutes to do a clone is a pretty long time to wait though.
- gpapilion 6y agoI’ve found as an industry we’ve moved to more complex tools, but haven’t built the expertise in them to truly engineer solutions using them. I think lots of organizations could find major optimizations, but it requires really learning about the technology you’re utilizing.
- smitty1e 6y agoThe whole point of being an Agile "generalizing specialist" is that one is a mile wide and an inch deep.
- gpapilion 6y agoWhich i think is a fair approach when you’re early on. When you have a dev efficiency team you’re no longer hiring generalists.
- temporallobe 6y agoThis, this, so much this. When we build more complexity into a system, the less we understand it, similar to how development frameworks create multiple layers of abstraction to the point where the developers have no idea what actual code the framework produces, much less how to fix it.
- slx26 6y agoYes, we probably need people to stop thinking about tools as if they "solved" problems; what they really do is "transform" them. Now instead of having to deal with the original problem, you only need to deal with part of it and part of the new problem of using the tool that's supposed to help you, plus any leaks you might have because tools rarely solve problems perfectly. It's a trade-off, and you need to be aware of these transformations.
- SamuelAdams 6y agoAnother way of looking at it is this is the current golden age of infosec. Think of all these complex systems developers and SysAdmins need to maintain at a company. Then think of how well each person knows each technology. Most of them will be "T" shaped, ie know one tech well but surface-level on all the others. If I know several tools really well (or better than the company's sysadmins / devs) I can probably find some security issues with them.
- mandeepj 6y agoLooks like Pinterest’s team is confused about Git Branches. These are not real full copy versions of the main branch like in SVN or TFS. A branch in Git world is simply a pointer to a specific commit in the code push history. Having said that, happy to be proven wrong, and learn about it.
- ekimekim 6y agoIIUC the issue here is the depth option - they're telling it to only fetch the last 50 commits, but they were fetching the last 50 commits from EACH branch. In other words, they were fetching all commits that are within 50 commits of any branch head. By restricting the branches, they drastically reduce the set of commits to fetch.
- chrisweekly 6y agoyeah,esp given 2500 branches
- tracer4201 6y agoI truly appreciate articles like this — it’s warming to see other companies running into the kinds of issues I’ve ran into or had to deal with, and more so that their culture openly discusses and shares these learnings with the broader community. The most effective organizations I’ve worked at built mechanisms and processes to disseminate these kinds of learnings and have regular brown bags on how a particular problem was solved or how others can apply their lessons. Keep it up Pinterest engineering folks.
- lerpapoo 6y agotldr can i guess it was doing some extra network roundtrips or something?
- segfaultbuserr 6y agoBetter title: A one-line change decreased our "git clone" times by 99%. It's a bit misleading to use "build time" to describe this improvement, as it makes people think about build systems, compilers, header files, or cache. On the other hand, the alternative title is descriptive and helpful to all developers, not only just builders - people who simply need to clone a branch from a large repository can benefit from this tip as well.
- CGamesPlay 6y agoThey set out to reduce build times, not to reduce git checkout times. It turns out that 99% of the entire build was spent downloading code.
- lytedev 6y agoThis isn't true either, as the article says that builds went from 40 minutes to 30 minutes. The time spent cloning was presumably about 10 minutes and came down very far, presumably by 99%.
- phildenhoff 6y agoIt says from 40 minutes to 30 seconds, not minutes.
- Thorrez 6y ago> the article says that builds went from 40 minutes to 30 minutes. Where in the article does it say that? The article says this: > This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result. Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds. Both of those sentences say the clone time was reduced by 99%. There are percentage numbers given for how much the build time was reduced, nor any numbers about the total build time.
- lytedev 6y agoI stand quite corrected. Sorry, all!
- chinhodado 6y agoWhen I first joined one of my previous jobs, the build process had a checkout stage where it was blowing away the git folder and checked out from scratch the whole repo every time (!). Since the build machine was reserved for that build job I simply made some changes to do git clean -dfx & git reset --hard & git checkout origin branch. It shaved off like 15 minutes of the build time, which was something like 50% of the total build time.
- mikepurvis 6y agoIt's frustrating how many ways there are for a git clone to get out of sync, especially when it's an automation-managed one that is supposed to be long-lived (think stuff like gracefully handling force-pushed branches and tags that are deleted). I've dealt with a bit of this with my company's Hound (code search engine) instance. Currently there's a big snarl of fallback logic in there that tries a shallow clone, but then unshallows and pulls refs if it can't find what it's looking for, culminating in this ridiculousness: git fetch --prune --no-tags --depth 1 origin+{ref}:remotes/origin/{ref} See the whole thing here: https://github.com/mikepurvis/hound/blob/6b0b44db489f9aeff390ad8a95cc57d6543b002e/vcs/git.go#L60 https://github.com/mikepurvis/hound/blob/6b0b44db489f9aeff39... The pipeline I manage is many repos rather than a monorepo, and maintaining long-licheckouts in this context is not really realistic, but what does work and is very fast is just grabbing tarballs— GitLab and Github both cache them, so they don't don't cost additional compute after the first time, and downloading them is strictly less transfer and fewer round trips than the git protocol. The only real cost is that anything at build time which needs VCS info (eg, to embed it in the binary) will need an alternate path, for example having it be able to be passed in via an envvar.
- mixmastamyk 6y agoA new checkout is good practice. Using refspec and depth options can make it quick.
- deleted 6y ago[deleted]
- jniedrauer 6y agoOne of the (many) things that drives me batty about Jenkins is that there are two different ways to represent everything. These days the "declarative pipelines" style seems to be the first class citizen, but most of the documentation still shows the old way. I can't take the code in this example and compare it trivially to my pipelines because the exact same logic is represented in a completely different format. I wish they would just deprecate one or the other.
- sxp 6y ago> Even though we’re telling Git to do a shallow clone, to not fetch any tags, and to fetch the last 50 commits ... What is the reason for cloning 50 commits? Whenever I clone a repo off GitHub for a quick build and don't care about sending patches back, I always use --depth=1 to avoid any history or stale assets. Is there a reason to get more commits if you don't care about having a local copy of the history? Do automated build pipelines need more info?
- MarkSweep 6y agoI can’t speak for the original post, but I’ve seen other people[1] increase the commit count because part of the build process looks for a specific commit to checkout after cloning. If you have pull requests landing concurrently and you only clone the most recent commit, there is a race condition between when you queue the build with a specific commit id and when you start the clone. All that being said, I don’t know why you would need you build agents to clone the whole damn repo for every build. Why not keep a copy around? That’s what TFS does. One other thing I've seen to reduce the Git clone bottleneck is to clone from Git once, create a Git bundle from the clone, upload the bundle to cloud storage, and then have the subsequent steps use the bundle instead of cloning directly. See these two files for the .NET Runtime repo[2][3]. I assume they do this because the clone step is slow or unreliable and then the subsequent moving around of the bundle is faster and more reliable. It also makes every node get the exact same clone (they build on macOS, Windows, and Linux). Lastly, be careful with the depth option when cloning. It causes a higher CPU burden on the remote. You can see this in the console output when the remote says it is compressing objects. And if you subsequently do a normaly fetch after a shallow clone, you can cause the server to do ever more work[4]. 1: https://github.com/dotnet/runtime/pull/35109 https://github.com/dotnet/runtime/pull/35109 2: https://github.com/dotnet/runtime/blob/693c1f05188330e270b01a6bce8ad22ab7b33f23/eng/pipelines/common/checkout-job.yml https://github.com/dotnet/runtime/blob/693c1f05188330e270b01... 3: https://github.com/dotnet/runtime/blob/693c1f05188330e270b01a6bce8ad22ab7b33f23/eng/pipelines/common/clone-checkout-bundle-step.yml https://github.com/dotnet/runtime/blob/693c1f05188330e270b01... 4: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...
- 6y ago
- Scaevolus 6y agoFor CI on large repos, you can do much better than this by using a persistent git cache. It takes a little finessing to destroy it if it's corrupt and avoid concurrent modifications, but it's extremely worth it.
- hakre 6y agoyou mean syncing to git bares on CI-nodes and then in the build not using a WAN remote but just clone from the bare with hard-links and checkout?
- chrisweekly 6y agoI find the self-congratulatory tone in the post kind of off-putting, akin to "I saved 99% on my heating bill when I started closing doors and windows in the middle of winter." If your repos weigh in at 20GB in size, with 350k commits, subject to 60k pulls in a single day, having someone with half a devops clue take a look at what your Jenkinsfile is doing with git is not exactly rocket science or a needle in a haystack. (Here's hoping they discover branch pruning too; how many of those 2500 branches are active?) As a consultant I've seen plenty of apallingly poor workflows and practices, so this isn't all that remarkable... but for me the post seems kind of pointless.
- paledot 6y agoIndeed. I wasn't aware of that specific git option, but a build pipeline with a checkout step taking FORTY MINUTES is unacceptable. Plenty of ways to solve that problem, but it's a problem that never should have made it into a critical workflow. I don't care for casting stones. It's clearly a big win, and you don't get numbers like that every day. But I feel like someone should've twigged to this much sooner.
- deleted 6y ago[deleted]
- deleted 6y ago[deleted]
- AdamJacobMuller 6y agoI expected this to be some micro-optimization of moving a thing from taking 10 seconds to 100ms. > Cloning our largest repo, Pinboard went from 40 minutes to 30 seconds. This is both very impressive as well as very disheartening. If a process in my CI was taking 40 minutes I would be investigating sooner than a 40-minute delay. I don't mean to throw shade on the pintrest engineering team, but, it speaks to an institutional complacency with things like this. I'm sure everyone was happy when the clone took 1 second. I doubt anyone noticed when the clone took 1 minute. Someone probably started to notice when the clone took 5 minutes but didn't look. Someone probably tried to fix it when the clone was taking 10 minutes and failed. I wonder what 'institutional complacencies' we have. Problems we assume are unsolvable but are actually very trivial to solve.
- nemothekid 6y agoI'm not sure this is complacency - this just seems like regular old tech debt. The build takes 40 minutes but everyone has other things to do and there is no time to tend to the debt. Then one day someone has some cycles and discovers a one line change fixes the underlying issue. I'm sure many engineering projects have similar improvements that just get a ticket/issue opened and never revisited due to the mountain of other seemingly pressing issues. From IPO to the start of the year Pinterest stock price had been trending downwards - I'm sure there was more external pressure to increase profitability than to fix CI build times. The stock has completely turned around since COVID, so I'm sure that changes things
- mehrdadn 6y agoIMHO (from having addressed such CI issues personally on teams that otherwise wouldn't bother) it's likely due to other factors, like a lack of interest, being scared of breaking the build, not being terribly comfortable touching build scripts, or the inability to run scripts locally, than a genuine lack of time. The returns you can get can be ridiculously huge across the entire team compared to the hours you might spend, but I've found many people just aren't terribly interested in sitting down and digging into ugly scripts and pushing dozens of commits to figure out what might be slowing things down. And honestly, it's not exactly trivial to structure things in a way that's simultaneously both efficient and maintainable, especially if you're refactoring an existing system instead of starting from scratch, so that can be another turn-off.
- YokoZar 6y agoCan someone explain the intended meaning behind calling six different repositories "monorepos"? It sounds to me like you don't have a monorepo at all and instead have six repositories for six project areas.
- yen223 6y agoMy interpretation is that each "monorepo" is a big git repository that consists of a collection of individually-deployed services, as opposed to having a single git repository per service. I do not know whether that's what the blog author meant by that though.
- whatatita 6y agoI got that impression too. I can imagine the Pintrest monorepo, for example, has the website and server code together. Their iOS and Android repos may contain the code for multiple apps. Though, I'm not aware of which other apps Pintrest (the company) creates besides the obvious one.
- aidanhs 6y agoI sympathise a lot with this post! Git cloning can be shockingly slow. As a personal anecdote, clones of the Rust repository in CI used to be pretty slow, and on investigating we found out that one key problem was cloning the LLVM submodule (which Rust has a fork of). In the end we put in place a hack to download the tar.gz of our LLVM repo from github and just copy it in place of the submodule, rather than cloning it. [0] Also, as a counterpoint to some other comments in this thread - it's really easy to just shrug off CI getting slower. A few minutes here and there adds up. It was only because our CI would hard-fail after 3 hours that the infra team really started digging in (on this and other things) - had we left it, I suspect we might be at around 5 hours by now! Contributors want to do their work, not investigate "what does a git clone really do". p.s. our first take on this was to have the submodules cloned and stored in the CI cache, then use the rather neat `--reference` flag [1] to grab objects from this local cache when initialising the submodule - incrementally updating the CI cache was way cheaper than recloning each time. Sadly the CI provider wasn't great at handling multi-GB caches, so we went with the approach outlined above. [0] https://github.com/rust-lang/rust/blob/1.47.0/src/ci/init_repo.sh#L50-L68 https://github.com/rust-lang/rust/blob/1.47.0/src/ci/init_re... [1] https://github.com/rust-lang/rust/commit/0347ff58230af512c9521bdda7877b8bef9e9d34#diff-a14d83f2e928fc5906d026a42cb16f021b452709b88bc3fd85c63e741cbd9a42R70 https://github.com/rust-lang/rust/commit/0347ff58230af512c95...
- bertr4nd 6y ago> Contributors want to do their work, not investigate "what does a git clone really do". Exactly this. Especially if the repo and CI pipeline are complicated, it is incredibly easy to just assume “it’s slow” is a fact of life. And from the point of view of the dev-productivity team, well, they have tons of possible issues to deal with at any given time. Not just CI but the repos themselves, the build system, maybe IDEs, debuggers, ... Sure the fix ends up being easy but you have to know to go looking for it.
- IggleSniggle 6y agoWhen you’ve got a billion other tasks to do, you might even know that it could be orders of magnitude faster and still not fix it, simply because of higher priority work. Frankly, I’d rather spend extra time trying to address problems/bugs/potential security holes in the actual shipped code than in fixing a poorly working CI pipeline...and I’m the kind of dev who gets really irritated by these problems. But you have to prioritize. Basically, barring “external” forces like cost overflow, customer unhappiness, or similar...stuff like that gets fixed at an equilibrium point between how much the problem hurts the dev, how adjacent to the codebase the devs current work is, and how interesting/irritating the dev finds the problem.
- cma 6y ago> For Pinboard alone, we do more than 60K git pulls on business days. Can anyone explain this? Seems ripe for another 99% improvement even with hundreds of devs.
- altdatathrow 6y agoAn unhealthy obsession with CI/CD is the usual culprit.
- quickthrower2 6y agoParkinson's Law of builds. "work expands so as to fill the time available for its completion", or in this case the available time is the point at which people can't stand the build taking too long. 30-60 minutes is normal because anything > 1 minute required you to context-switch anyway, and > 60 minutes means you are now at risk of taking a day if you have a work queue of a 1-pizza team. So [1..60] range causes a grumble but nothing will be done.
- scoot_718 6y agoWho cares if all you're doing is optimizing the massive spam operation that is Pinterest? You've made garbage faster. Well done.
- hansdieter1337 6y agomissleading title. Not the build time was decreased by 99%. Only the git checkout step was.
- s9w 6y agoclickbait
- ibains 6y agoThus just shows how poor visibility into git is, I hope it gets better. Building a product with poor visibility and ridiculing users for not knowing internals is the worst practice in Computer Science. Hadoop did the same, and has set a record of fastest software to become legacy. Super nice to see great comments here and the nice article.
- wruza 6y agoIt is pinteresting that a webapp for making your image saving obsession easier to satisfy takes hundreds to thousands developer actions per day and repository sizes of tens of gigabytes.
- timzaman 6y agoMisleading title. They reduced their clone time by 99%. Not their build time.
- dastx 6y agoWith a repo that is 20GB, I can imagine that could be 99% of the build time.
- smsm42 6y agoTLDR: they reduced "git clone" time on their massive monorepo by making it only check out master branch when building in Jenkins.
- dusted 6y agoThis is the most I've ever gotten out of pinterest, other than this, it's just the "wrong site that google turns up, that I can't use because it wants me to create an account just to watch the image I searched for"
- randunel 6y agoThe most frequent search keyword that I use is "-pinterest"
- amelius 6y agoYes, there seems to be no way to make it clear to Google that we want to never see certain websites in our search results. Yet, Google claims they need our information to "improve our experience".
- mcv 6y agoIf Google wants my information to improve my experience, I'd love to be able to vote search results up or down. Or entire sites, like pinterest and content farms.
- hadrien01 6y agoWasn't that a thing? I remember a +1 button somewhere on Google Search Edit: I misremembered, it was a social network thing from Google+ https://www.techspot.com/news/43064-google-adds-1-button-to-search-results.html https://www.techspot.com/news/43064-google-adds-1-button-to-...
- rachelbythebay 6y agoThere was a thing called SearchWiki where you could adjust your own results. It didn't last long.
- andromeduck 6y ago-pintrest should be a search extension.
- andrelaszlo 6y agoWe recently reduced our build times by 5-10% or so by changing the default bcrypt iteration count (for tests). It also felt silly once we found it.
- nathan_f77 6y agoThanks so much for this tip! I just made this change and some of my tests are now much faster. Here's the result for one of the affected tests (averaged across 5 runs): Before: 1.39 seconds After: 0.62 seconds I have this default line in config/initializers/devise.rb: config.stretches = Rails.env.test? ? 1 : 11 So hashing user passwords was already very fast. But I'm also manually calling BCrypt in some other places, so these calls are now much faster as well.
- mytailorisrich 6y agoOne thing to keep in mind is that this obviously changes the timing of your software with respect to production behaviour, which may or may not matter depending on what you are testing.
- user5994461 6y agoYou should consider doing the same thing in production. It's a trope at this point how the modern slow hashing algorithms are utterly misconfigured. Stopped counting how many times I've seen it. Take a whole second to compute a hash on the production machine because "hashing is supposed to be slow", noting the production server is a low frequency Xeon that has many core but they're half as slow as your development with a 4GHz i7-9999. Hashing is supposed to take milliseconds, not seconds. If it's taking longer than 100 ms you need to make it faster. edit: found the problem, this bad stackoverflow answer that's been spreading bad recommendations for years https://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt/83382#83382 https://security.stackexchange.com/questions/17207/recommend...
- nathan_f77 6y agoIs there any way to do this for GitLab CI [1]? I'm using GIT_DEPTH=1, but I'm not sure how to set refspecs. It's not too important right now since it only takes about 11 seconds to clone the git repo, but maybe it's a quick win as well. [1] https://docs.gitlab.com/ee/ci/large_repositories/ https://docs.gitlab.com/ee/ci/large_repositories/
- mcintyre1994 6y agoThe docs seem to give the impression that they already do this, but it'd be great if someone from Gitlab could confirm because it doesn't use the refspec term or show the resulting git command. > The following example makes the runner shallow clone to fetch only a given branch; it does not fetch any other branches nor tags. https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cloning https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cl...
- paytonburd 6y agoCheckout the docs here https://docs.gitlab.com/ee/ci/yaml/README.html#git-fetch-extra-flags https://docs.gitlab.com/ee/ci/yaml/README.html#git-fetch-ext... GIT_FETCH_EXTRA_FLAGS accepts all options of the git fetch command
- mcintyre1994 6y agoThanks for that link - seems to be further evidence they do already do this refspec flag by default. > The default flags are: GIT_DEPTH. The list of refspecs. A remote called origin.
- yoz-y 6y agoBecause of strife with 99% claim. If the pull time took 39.9min (and thus build took 0.1min = 6sec) then a 99% decrease in pull time would result in 99% decrease of total time and you would get 30sec total time in the end. (Rounding to 0 decimal places). Not that any of this is important for the article to be interesting. In a previous job we had to fight long pull times and we quickly created a git repo for CI that would sit on a machine next to the CI server and would periodically pull from GitHub to avoid the CI to do pulls over Internet.
- sambe 6y agoOk, I'll ask the obvious question: why did setting the branches option to master not already do this? EDIT https://www.jenkins.io/doc/pipeline/steps/workflow-scm-step/ https://www.jenkins.io/doc/pipeline/steps/workflow-scm-step/ makes it sounds like the branches option specifies which branches to monitor for changes, after which all branches are fetched. This still seems like a counter-intuitive design that doesn't fit the most common cases.
- ar7hur 6y agoAlternative title: "How one line of code made our build time 100x what it should have been"
- saagarjha 6y agoSeems like there's a lot of hostility towards the title, which might be considered the engineering blog equivalent of clickbait. If the authors are around, the post was quite informative and interesting to read, but I'm sure it would have been much more palatable with a more descriptive title. But back on topic: does anyone have any insight into when git fetches things, and what it chooses to grab? It is just "when we were writing git we chose these things as being useful to have a 'please update things before running this command' implicitly run before them"? For example, git pull seems to run a fetch for you, etc.
- inopinatus 6y agoMy CI servers have to build branches as well, though. A fresh clone for every build? No wonder it was slow, but even this solution seems inefficient. My preferred general solution is a persistent repository clone per build host, maintained by incremental fetch, and use git worktree add, not git clone, to checkout each build.
- nabaraz 6y agoIs it a common practice to clone the repo on every build (especially on web apps)? I just have Jenkins navigate to an app folder, run few git commands (hard reset, pull), and build (webpack).
- mister_hn 6y agoI'm not impressed by the author of the post, since it's also something documented in the plugin, saying that you should not checkout all the branches, if not interested. The default behaviour of course is to get all of them.
- Dylan16807 6y agoWell, good advice, and good for them, but > Cloning monorepos that have a lot of code and history is time consuming, and we need to do it frequently throughout the day in our continuous integration pipelines. No you don't! If removing per-build clones was the only way to speed things up, I'm absolutely sure you could figure out how with medium difficulty at most.
- mcv 6y agoOn my first job, 20 years ago, we used a custom Visual C framework that generated one huge .h file that connected all sorts of stuff together. Amongst other things, that .h file contained a list of 10,000 const uints, which were included in every file, and compiled in every file. Compiling that project took hours. At some point I wrote a script that changed all those const uints to #define, which cut our build time to a much more manageable half hour. Project lead called it the biggest productivity improvement in the project; now we could build over lunch instead of over the weekend. If there's a step in your build pipeline that takes an unreasonable amount of time, it's worth checking why. In my current project, the slowest part of our build pipeline is the Cypress tests. (They're also the most unreliable part.)
- ravishi 6y agoAt my second job in the industry I worked on a Python project that had to be deployed in a kind of sandboxed production environment where we had no internet access. Deploys were painful, as any missing dependency had to be searched in our notebooks over 3G, then copied to an external storage, then plugged into a Windows machine, uploaded to the production server through SCP and then deployed manually over SSH. Sometimes we spent hours doing this again and again until all dependencies were finally resolved. I worked there for almost a year, did many cool gigs and learned a lot. But my most valuable contribution came when at some point, tired of that unpredictable torture that were the deploys, started researching into solutions. I set up a pypi proxy into one of our spare office machines and routed all my daily package installs through that. Then I copied that entire proxy content into the production machine before every deploy, and voila, no more surprises. I left this job a few weeks later, but have heard that this solution was very useful for many devs that joined the team afterwards.
- leothekim 6y ago“We have six main repositories at Pinterest: Pinboard, Optimus, Cosmos, Magnus, iOS, and Android. Each one is a monorepo and houses a large collection of language-specific services.” What is an “iOS monorepo” supposed to be like?
- dada78641 6y agoThis reminds me of my first programming job in 2005, working with Macromedia Flash. They had one other Flash programmer who only worked there every once in a while because he was actually studying in college, and he was working on some kind of project from hell that, among other problems, took about two minutes to build to SWF. Eventually they stopped asking him to come because he couldn't get anything done, and so I had a look at it. In the Movie Clip library of the project I found he had an empty text field somewhere that was configured to include a copy of almost the entire Unicode range, including thousands of CJK characters, so each time you built the SWF it would collect and compress numerous different scripts from different fonts as vectors for use by the program. And it wasn't even being used by anything. Once I removed that one empty text field, builds went down to about ~3 seconds.
- tetris11 6y agoI take it that this is not something he added himself, but was likely a catch-all default of textfields at the time?
- omegote 6y agoYep. In order to use non-standard fonts in Flash I recall you had to embed the fonts, even if the movie clip containing the textfield was not being used anywhere.
- SamuelAdams 6y ago> In the case of Pinboard, that operation would be fetching more than 2,500 branches. Ok, I'll ask: why does a single repository have over 2,500 branches? Why not delete the ones you no longer use?
- casperb 6y agoProbably because they have 1600 employees and the 2500 branches are the active ones.
- tetris11 6y agoIf you have several releases with different targets, and want to make future security updates accessible to all
- DoingIsLearning 6y agoThey could already be doing that. That is if we assume they copy google's philosophy of a single monolith repository. Pinterest has about 2000 employees, assuming 20% are active developers thats about 400 people, that gives you roughly 6 branches per developer which wouldn't be outrageous.
- dahfizz 6y agoWhere I work doesn't delete branches, because there is no reason to. Git branches have essentially zero overhead and deleting them is just extra complexity in the CI toolchain. Deleting branches also deletes context in some scenarios. When dealing with an old codebase its nice to be able to checkout the exact version of the code at some point without having to dig through the log to get hashes and then dealing with a detached head. The example in the article is a bit of a special case. It is a huge, and old, monorepo. In the typical case, fetching everything and fetching master is equivalent because all commits in all branches make their way into master anyway. If you have a weird branching strategy where you maintain multiple, significantly diverged branches at once, but only care about one of those branches at build time, then this optimization would save you time.
- richardwhiuk 6y ago
- jakub_g 6y agoSemi-related for JS developers: if you do `eslint` as part of your build, make sure `node_modules` (and `node_modules` in subolders if you have monorepo-ish solution) is excluded.
- joshribakoff 6y agoMisleading title. They reduced git clone time 99%, not build times.
- JoeAltmaier 6y agoSo git doesn't scale well with wide, deep source histories? That's a failing of git I think, not the Engineers who may even have written that line when the source base was far less gnarly.
- bluedino 6y agoMy similar story goes like this: We had CRM software that let you setup user defined menu options. Someone at our organization decided to make a set of nested menu options where you could configure a product, with every possible combination being assigned a value! So if you had a large, blue second generation widget with a foo accessory and option buzz, you were value 30202, and if was the same one except red, it was 26420... Every time the CRM software started up, it cycled through the options, generated a new XML file with all the results, this took about a minute and created like a 60MB file. The fix was to basically version the XML file and the options definition file. If someone had already generated that file, just load the XML file instead of parsing and looping through the options file. Started up in 5 seconds! What was the excuse that it took so long in the first place? "The CRM software is written in Java, so it's slow."
- soulofmischief 6y agoThe title is a bit of a misnomer, isn't it? > This simple one line change reduced our clone times by 99% and significantly reduced our build times as a result. Sounds like it didn't reduce build times quite by 99%.
- fortran77 6y agoI'd rather Pinterest increased their build times by 99% so they could do less damage to search results.
- cj 6y agoI once reduced the speed of our test suite from 10 mins to < 5 minutes by changing 2 characters in 1 line... Then bcrypt work factor! It was originally 12, reduced it to 1 (don’t worry, production is still 12)
- muststopmyths 6y agoI'm a git noob, so I'm sorry if this sounds dumb but wouldn't git clone --single-branch achieve the same thing (i.e, check out only the branch you want to build) ? Also, why would you not only check out one branch when doing CI ?
- jxramos 6y agoLooks like its implied from the documentation Implies --single-branch https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt https://git-scm.com/docs/git-clone#Documentation/git-clone.t...
- muststopmyths 6y agohmm, the --depth implies single-branch, but the +refs overrode it by making sure it had the data to match all branches because of the wildcard refspec ?
- uglycoyote 6y agoHe says that "Pinboard has more than 350K commits and is 20GB in size when cloned fully." I'm not clear though, exactly what "cloned fully" means in context of the unoptimized/optimized situation. He says it went from 40 minutes to 30 seconds. Does this mean they found a way to grab the whole 20GB repo in 30 seconds? seems pretty darn fast to grab 20GB, but maybe on fast internal networks? Or maybe they meant that it was 20GB if you grabbed all of the many thousands of garbage branches, when Jenkins really only needed to test "master", and finding a solution that allowed them to only grab what they needed made things faster. I'm also curious about the incremental vs "cloning fully" aspect of it. Does each run of Jenkins clone the repo from scratch or does it incrementally pull into a directory where it has been cloned before? I could see how in a cloning-from-scratch situation the burden of cloning every branch that ever existed would be large, whereas incrementally I would think it wouldn't matter that much.
- degrews 6y ago> He says that "Pinboard has more than 350K commits and is 20GB in size when cloned fully." I'm not clear though, exactly what "cloned fully" means in context of the unoptimized/optimized situation. It probably means including all commits. It looks like they were successfully only pulling the last 50 commits, but they were doing that for each of 2500 branches. Now they are pulling only the most recent 50 commits for one branch.
- csours 6y agoTroubleshooting CI/CD feels like troubleshooting a printer: What the hell is it doing now and why is it doing that?!
- KnobbleMcKnees 6y agoThe article is erroneous in many ways as others have described, but the main error I see is that it says 'git clone' is run before the fetch. It should be 'git init'