24 ms·

Experiment: The costs of slow build times

167 points by pimterry 4y ago

yosito 4y ago
> The Bottom Line > It’s cheaper—and less frustrating for your developers—to pay more for better hardware to keep your team on track. I agree with this point, 100%. And not just in the context of build hardware. It's often cheaper to pay for specialized services rather than paying in-house developers to do those things. I've lost count of the number of times in my career that I've spent days of effort (at $200/hr) so that the company I was working for could avoid paying for some $15/mo SaaS.
- deleted 4y ago
  [deleted]
- emerongi 4y ago
  Premature optimization is the root of all evil. It's almost always better to start by using a specialized service, even if it's overpriced. Later on you can evaluate much more adequately if you'd save money/time by building your own.
  - gpderetta 4y ago
    Premature "Premature optimization is the root of all evil" is the root of all evil. Recurse as needed.
- EdwardDiego 4y ago
  I'm experiencing the contrary, my company paying over a 100K a year for a managed service I could deploy and maintain single handed and it would still require less than 5% of my time. But you know, when you're scared of the unknown unknowns, the risk, managed services are attractive.
  - ilyt 4y ago
    If we started from scratch, all the automation we have would take months to build so cloud might be cost effective for first few years. But with automation already in place cost to both maintain that and that once-a-month visit to replace a hard disk in datacenter, or once-a year half-day to install few servers is far cheaper than anything cloud, especially if you need bandwidth
  - jameshart 4y ago
    The cost to a business of lost developer productivity is not the salary cost of the developer doing non dev work. It is the lost productivity of the developer not making the things you’re paying them for. The reason the business is willing to spend $x a year on the developer is because the developer produces >$x a year of value, in the form of revenue generating software. If you divert 5% of your time into doing something else, you are taking money the business wanted to invest into capital (new software features) and spending it on… something else. Lunch orders or whatever. Not making capital, anyhow.
    - kqr 4y ago
      Huh. I find myself partly disagreeing with this but I cannot for the life of me articulate why. Upvoted in the hope that a discussion is sparked...
  - TYMorningCoffee 4y ago
    There also qualitative benefits like the knowledge the developer and the company gains. Also, the developer might enjoy the change of scenery.
- plonk 4y ago
  For us the work was mostly in adapting the old codebase to CI/CD and writing/debugging the workflows. There’s almost no additional cost to running self-hosted runners, in fact we wouldn’t be using GitHub at all if they didn’t offer a local runner.
- albrewer 4y ago
  My entire role at my current company could be better served by spending $60k on software licenses a year, but they choose to pay triple that for me to create a shittier version of it. I'm not complaining, really, because it's been an enormous amount of fun and I've been able to solve a ton of interesting problems, but from a business perspective it makes absolutely no sense.
tjpnz 4y ago
Are there really people who still need convincing of this? At the same time you probably shouldn't be throwing more hardware at the problem without trying to understand it first. I've witnessed numerous times throughout my career what can be saved by tuning a few parameters.
- papito 4y ago
  It's the resource curse. Why think about it too much if you can just throw resources at the problem? This is not just an issue with builds. Seems like developers treat everything as "free", from HTTP round-trips to database queries. Not to mention the tendency to craft convoluted and heavy solutions to rather standard problems. Because "this is how it's done now". Standing here, holding my hammer. Lo, on the horizon: a nail!
- KronisLV 4y ago
  > Are there really people who still need convincing of this? Some might just not care that much, or nobody might speak up about the problem, not wanting to make it seem like they're just complaining, in the case of nobody being able to provision more resources, or optimize the build. For example, in some environments, getting a better CI node might be associated with red tape, lots of back and forth with a different department or something like that, especially if it's all on-prem instead of using managed services or cloud resources. > At the same time you probably shouldn't be throwing more hardware at the problem without trying to understand it first. I've witnessed numerous times throughout my career what can be saved by tuning a few parameters. This is an excellent point! In a context of containers (just an example, though illustrates some pitfalls nicely) there's a world of difference between using a base image that has the runtime and tools you need, vs installing them every time. There's a lot of difference between pulling in your dependencies every time, vs using a cache for it or an instance of Nexus that acts as a caching proxy (though that's not very nice to configure in most languages). There's even difference between making sure that packages install correctly from the cache, vs having them already be installed in some intermediate layer that you can just reuse when building your container. Even without containers, there can be lots of things to take into consideration, like whether you could build your back end and front end (in the context of webdev) or other components in parallel, whether you could run any tests you might have in parallel, or whether there's not duplicated work going on in there somewhere.
carreau 4y ago
I my experience if you get more builders/faster hardware someone higher up will end up asking for more items in the build-matrix, and the CI time will balloon up again. It's rarely only a question of just allocating money to more/better hardware, it's also a question of policy and willingness of your organisation to keep CI time short/feedback fast.
- substation13 4y ago
  The solution is a build system that scales by caching between runs - most PRs only change a few files, after all.
  - carreau 4y ago
    As I replied to the sibling comment, yes, I agree, but caching is not magic. You need to know what to invalidate, and you need to be allowed to spend time making test group that run depending on what have changed. So without policy and having devs being allowed to work on optimizing CI/reducing time, you can list plenty of strategies to make it faster but it will likely not happen.
    - ohgodplsno 4y ago
      Caching is pretty damn simple with any sensible build tool that simply stores hashes of files and modules. Caching in the JVM ecosystem is pretty much free, for example.
      - carreau 4y ago
        Sure, but you seem to be read this blog post in the narrow view of building as only "compiling an artifact" and not running the test suite across many platforms/condition, or any other work. I mean the author even points out that building the kernel is only a use-case that takes time for measurement purposes. For some projects I work on the compilation itself takes maybe 1/10 max of the total build and test. So even with infinite agressive compilation caching I would not gain much.
Hardwired8976 4y ago
I worked from home on Wednesday and I managed to watch one episode of Better Call Saul while waiting for the compilation of my code lol.
- titzer 4y ago
  Chrome dev?
  - endgame 4y ago
    Nah, Chrome is larger than that. I built NixOS 22.11 from source and Chromium took over 6 hours on a Ryzen 7 3800XT with 32GB of RAM. Had to run the build on spinning rust though, because neither my SSD and certainly not my tmpfs were large enough.
  - Hardwired8976 4y ago
    Cobol, the crazy thing is I gotta wait the 5min before I can even debug it as we aren't allowed to run it locally
    - pharmakom 4y ago
      Someone should pause on the Better Call Saul and write some Bazel rules for COBOL.
tjoff 4y ago
This is also a big reason to self-host. Without this beta github offer: 2-core CPU (x86_64), 7 GB of RAM, 14 GB of SSD space Your last development machine that was thrown away is so much faster than this (and we do have great tools for administer stuff like this nowadays). Hell, the computer I bought as a student in 2008 is comparable(!) (though it didn't have an SSD). And it will have so much better network connectivity with everything else on your network. Whenever you hear cloud, realize that the dedicated time you will get (unless you specifically pay for it (in which case it will be more expensive than self-host)) is in the same order the phone you had two generations ago. That is why gmail search sucks. Because they can't afford to really search your messages and can not even do exact matching properly. So yes, apparently github is fixing this now and if paying for this problem makes sense for you do it. But this is a problem that was partly invented by the cloud in the first place.
- qersist3nce 4y ago
  >Your last development machine that was thrown away is so much faster than this >the computer I bought as a student in 2008 is comparable >the phone you had two generations ago Good for Americans (or first-world citizens)
  - danuker 4y ago
    As a Romanian web dev, my 8-y-o laptop has a 4-core CPU, 12GB RAM, and 256GB of SSD space. The laptop cost about 110% of my monthly entry-level salary.
    - 5e92cb50239222b 4y ago
      Romania is significantly better developed and more affluent than most of the world. My similarly aged laptop had much worse hardware than what you're describing and cost 200%+ of my old entry-level salary. (I don't use laptops anymore.) GitHub CI also has much better network connectivity (both speed and especially ping) compared to most parts of my country.
  - ddorian43 4y ago
rpigab 4y ago
That's why you should only hire contractors, and shouldn't pay them while a build is running. Saves more money that way, and you don't have to care much about build duration.
- panzerboiler 4y ago
  Hallo. I am a contractor. Why you shouldn't pay for the time my machine is building your project?
  - wruza 4y ago
    Cause you can work on other projects meanwhile, multitasking is easy and never helps making mistakes. It’s also easy to contract on your terms. You just say “I want that for very cheap” and get flooded with interested contractors. /s
- DocTomoe 4y ago
  Good luck finding contractors who will agree to these terms. Waiting for built times is dead time which is filled with other activity, like documentation, or reading through the next requirement. What you will achieve with this policy is minimizing the number of build processes, from a few times a day to once or twice a week (or less), which is exactly what you do not want from a CI/CD / DevOps environment.
  - randomdata 4y ago
    Besides, you're not paying for one's time. You're paying for one's availability. The difference is subtle, but significant.
- chronial 4y ago
  Is this sarcasm about our absurd reality of hyper-capitalism or are you serious?
  - rpigab 4y ago
    I'm dead serious, we have to embrace the gig economy and everything it has to change in our lives for the better. Why would I be paying for bathroom breaks and idle time? I wish there was software to count how much time is lost when users minimize all windows or switch to desktop, so I could deduct it from their pay. I know some of this idle time is spent thinking about problems to solve related to the job, but it's something that can be done in your free time, or while commuting.
iLoveOncall 4y ago
> In short, if a developer is waiting on a build to run for one hour and doing nothing in that timeframe This should be at the very top of the article. It basically negates entirely the whole experiment because no developer will just sit and wait for hours for a build (unless they planned on taking that break anyway). The cost is not 0 but it is absolutely negligible.
- bombolo 4y ago
  https://xkcd.com/303/ https://xkcd.com/303/
- Aeolun 4y ago
  I don’t know what work you do, but if my test takes longer than 30 minutes I’ll start working on something else, and when I come back to fix whatever failed in the test (when I get to it a few hours later), I’ll have to reconstruct the entire problem space in my head again.
  - iLoveOncall 4y ago
    If your build time or test runtime takes 30 minutes I'd argue your problem won't be solved by faster hardware.
- tener 4y ago
  This requires a context switch which is expensive in it's own way. Read the article, it covers this pattern pretty well.
  - bstpierre 4y ago
    Even with the high end machine it still takes 27 minutes. If I have a 27 minute build I am absolutely going to context switch. If it’s more than about 3 minutes I am probably going to context switch. An interesting experiment would be to find the switch-time threshold for a wide range of developers. Typically I don’t rely on a full build for getting things accomplished, and not CI if I can help it. Local, incremental, and minimal path to feedback is what I aim for.
e1g 4y ago
I recently moved our GutHub CI to a self-hosted runner and it reduced CI times for all jobs by ~5x. One frustrating part is that a single GitHub runner can handle one job at a time, and our repos have 4-8 highly parallelized jobs. So we need multiple runners. To do this in an economic way, I made a Docker image and run 10 instances of it a single dedicated host from Hetzner. For ~$50/mo, we have unlimited CI and job run as fast as theoretically possible. The CI box has no inbound ports open to the internet, and the OS is set to auto-update, so the maintenance burden is low.
- Delemono 4y ago
  That sounds more like expected than an issue. You might not want to overload one instance. And running it like you do, is a no brainer anyway
  - e1g 4y ago
    Executing a GitHub runner in a container is a no-brainer, but it's still easy to overload the _host_ with too many jobs/containers, so managing resources is always up to the operator. I understand why GitHub has this constraint - to avoid clashes between jobs. If multiple jobs (from the same repository) are executing simultaneously within the same stateful environment, they are more likely to clash over shared resources (/tmp, cleanup tasks, database names, etc.). However, even if my jobs are clean and idempotent, GitHub is nudging me to think about runners as "VMs" rather than "containers" (because often CI jobs involve their own containers, and docker-in-docker is a pain), and "self-hosting a bunch of CI VMs" becomes expensive rather quick.
    - naikrovek 4y ago
      well, GHA supports running actions in containers, which is the best way to control the environment your CI/CD runs in, so putting your runner in a container won't work for everyone. but, if it does work for someone, doing what you've done will give a much better experience to their developers. my employer uses single-user VMs for it's runners; it works well, but sometimes actions invocation is high and it can take a few minutes for a runner to come around to taking my job. that would be much less of a problem with dockerized runners.
malinens 4y ago
One of the main metric for build time for me is to make it fast enough so you do not want switch to other task at all
- Cthulhu_ 4y ago
  Same, but this is why I will never even consider waiting for a CI pipeline; even if it's fast, there's always overhead in waiting for the pipeline to start, setting up the runners, cloning git, transferring files, etc. I always see it as a hierarchy; editor, local runtime, pre-commit / pre-push hooks, CI. At the moment, if this setup works, my pre-push hook should only run various tools (linters, unit tests) on the files that were affected by the commit. I'll leave it to CI to run the full suite of verifications and end-to-end tests.
- alkonaut 4y ago
  Devs need quick local builds to not lose their train of thought. But even when a build takes 5 seconds, a test suite might take 5 hours. CI is usually dominated by tests (and other non-compilation tasks such as packaging, static analysis, obfuscation, signing, even). Local builds can often be incremental and finish in seconds even if a full build takes ten minutes. My CI is basically 5% git checkout (varies), 2% build (a few minutes), 80% run tests, 5% static analysis, the rest is misc crap like uploads of artifacts, code signing, etc.
marginalia_nu 4y ago
Excepting C++ development, is slow build times really a hardware problem? CI software in general is very slow. But even building locally, there are many projects that have just ridiculous build times because it's just not a priority at all. I've seen small-medium sized Java projects with a 25 minute test suite. This has been in part due to extremely inefficient tests that keep assembling the entire system over and over each test, often with extreme redundancy in coverage, ensuring you need to keep running the entire suite whenever a test breaks. Nobody is going to run tests for half an hour unless you hold a gun to their head. So you make your change by 10 AM, usually something trivial, and then you spend the afternoon fixing the 30 tests that broke, running the suite over and over to find more broken tests to fix. It's almost comically inefficient.
- Aeolun 4y ago
  I mean, you spending the afternoon fixing all the tests that broke is a lot more efficient than you spending the night fixing those same issues when it gets deployed to production.
  - marginalia_nu 4y ago
    Sure, but I'm not convinced this necessary or sufficient to prevent that. It's rarely obvious what these types of tests actually test. It's like... just a bit of everything for the sake of good coverage. That and the inertia from having to wait half an hour for the test results makes it very compelling to start just "fixing" the tests by changing the assertions to expect the new value. If you have a more sensible testing strategy, it's obvious what is being tested (and thus also relatively clear what broke), and the turnaround is seconds possibly low single-digit minutes, so you don't have to rebuild the mental context every time you run the tests.
    - newaccount74 4y ago
      > "fixing" the tests by changing the assertions to the new value Oh my... I spent hours trying to figure out why I broke a test, only to figure out that my change actually fixed incorrect behavior! The previous dev who worked on the code had "fixed" the test failure by changing the assertion to expect the incorrect behavior! Automatic test suites with thousands of tests sound great in theory, but in practice it's not all rosy...
kunley 4y ago
Hmm, in addition to that, what about un-bloating the typical CICD to speed up the build?
- Aeolun 4y ago
  What part would you like to unbloat? Github is pretty light as far as I’m concerned.
tikkabhuna 4y ago
We use GitLab CI and solve this with tags. We have runners allocated with set resources and tagged with a t-shirt size. Developers can then tag their job based on this size. Single threaded jobs (curl-uploading a file) can be tagged "small" to get 1 CPU core. Building projects can be medium or large based on how big they are and developers can choose whether the large runner is needed. Because its simple to switch tags, developers can create a feature branch to swap the tag and observe how long it takes. All our runners are self hosted. There are fewer large runners, so developers are inclined to go small. Hypothetically, you can run all your jobs in large but then you may have to wait for availability but in practice we haven't seen any real contention. We run over 15,000 jobs a week on 6 physical servers with this pattern.
deleted 4y ago
[deleted]
pron 4y ago
Focusing on the cost side of software development is not very convincing for managers who need to make purchasing decisions. I believe most software is a high-margin product because manufacturing costs are close to zero. Even a relatively high difference in cost is not as much as a small difference in value. The one cost-side metric that has a big effect on value is time-to-market. What could be more convincing is the effect on product value that build times could have, and they certainly could (and I, for one, believe that they do, especially if they can be made to be less than 15 seconds or so). For example, it's certainly conceivable that very short build times make it easier to write more tests, which could result in more correct software with better features and/or a shorter time-to-market. Some products targeted at developers also make the mistake of focusing their marketing message on the cost, perhaps because that's the part of the business that developers personally feel, but it's not the message that would convince their employers.
- DrScientist 4y ago
  Totally agree. What matters is development velocity and quality - these are the things you will be competing on - not really cost of production. Slow builds impact both of the above of course.
- chasinglogic 4y ago
  Definitely, it's also important to remember how salaries vs. infrastructure costs show up on a P&L sheet. Salaries are going to be considered largely immovable and they won't "go down" or show as a lower number on the P&L as a result of your devs getting some time back. Whereas your infrastructure costs jumping up (even by a small amount, esp if you have any amount of scale) will set of alarms, rightly IMO, in the Finance department.
  - pron 4y ago
    Yep. In developer-targeted products the users are usually not the customers. Making the product appealing to users is (often) necessary, but it's insufficient to convince the actual customer. Sometimes the problem is in the product itself, but often it's just a matter of shaping the marketing message.
sluongng 4y ago
This is also a reason why many large organizations recently started to adopt a matured build tool solution such as: Bazel / Pants / Buck. They did the math.
- moffkalast 4y ago
  But did they also do the monster math?
  - EdwardDiego 4y ago
    It was a Bazel smash.
alkonaut 4y ago
Build is one of those things that I can't see the economy in buying cloud VMs for if you have lots of load. Buy a rack and a dozen cheap-ish rack chassis with some middle range CPUs with lots of cores but not necessarily the highest specs otherwise (Last gen we used 9700K and for a recent refresh it's AMD 7700X). You can get quite far at $1k per node and $12k for a rack. Then you can run 3-4 build agents per node so you have a 48 agent cluster in your rack. Electricity and management will be on top of the purchase price of course, but it's still a bargain. Even if you do use cloud compute for builds, it's worth having some self-hosted nodes for the base need and just using the cloud for peak use scaling.
- hhh 4y ago
  The cost of someone to manage and support this in my org would cost more than it is worth. Gitlab runners on an autoscaling group handles load for me and scales up and down with need. Took 2-3 hours to set up, and has lasted 2 years without incident. The value for me is cost (especially in time) to implement, mental burden, and risk of delaying projects.
  - alkonaut 4y ago
    It depends on what resources you have on hand obviously, but perhaps more it will depend on resource usage. There is little point buying a rack and having it idle. If you have very bursty workloads then scaling pays off. If you run your rick nice and warm even over nights and weekends then it's more economic to have self hosted I imagine.
ttyyzz 4y ago
As a Developer, build times sometimes are a blessing :D But thats beside the point. In my JavaScript frontend Framework / Nodejs backend world, build times are often times horrific due to the Single-threaded nature of JS. Hardware is often times not to blame.
- M4v3R 4y ago
  It really depends on which bundler/framework you use. We're developing a medium-sized Next.js 12 app recently and the build times are pretty good, the production build on my machine takes around 40 seconds.
- moffkalast 4y ago
  Gotta love getting paid for sitting around doing nothing :P
- Cthulhu_ 4y ago
  That's rapidly changing thankfully with newer tooling like Deno, Vite, etc; tools that are written in a native language (rust) instead of JS. I don't think it's to blame on the single-threadedness though, I've found that i/o is more often a bottleneck, thanks to the Node ecosystem's focus on many small libraries. I've seen improved performance in that regard in Yarn 2/3 though, since it stores dependencies in .zip files that are kinder on most storage mediums and it keeps a central repository instead of duplicating dependencies across projects.
avodonosov 4y ago
If you want to speedup build of a large Maven project, check https://github.com/avodonosov/hashver-maven-plugin https://github.com/avodonosov/hashver-maven-plugin
jurmous 4y ago
[deleted by author]
- skrebbel 4y ago
  Please stop
  - jurmous 4y ago
    Ok. Was the first time I tried this on a post and found it an interesting take so wanted to share but wanted to be transparent it was AI generated. But maybe others are doing it too much already which I am not aware of yet.
nottorp 4y ago
Isn't the real problem that no one does partial local builds any more? And maybe the lack of a local testing environment. Build servers are fine for the main branch, but why does a dev have to build the whole thing to quickly check the last 50 lines that they've written?
- amtamt 4y ago
  So many times it is not even 50, but 1 line. and those changes fail too. Also, more hardware does not make builds faster magically all the times. one needs to spend time to actually use more/ better hardware. Finally, build load comes in spikes. end of the working day, you have lots of builds all of a sudden. Few runners get overloaded in peak or many runners waste a lot of money. May be we need builds being done in some other timezones where its off-peak time, and even better, if people could pool their resources (and thats what SaaS runners do for us.)
- treis 4y ago
  >Isn't the real problem that no one does partial local builds any more? And maybe the lack of a local testing environment. I die a little inside every time my build is waiting for a runner while my M1 Mac is just sitting there.
  - mdaniel 4y ago
    AFAIK there's nothing stopping you from just spinning one up if you have the perms to the repo, although because of both your M1 and "untrusted code" reasons you'd likely want to put it in a virtual machine: https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository https://docs.github.com/en/actions/hosting-your-own-runners/... GitLab behaves similarly
- chillfox 4y ago
  Because we decided for some reason that CI builds (the build config) should not be able to be run locally for testing.
- bluGill 4y ago
comprev 4y ago
We use Github Actions on self-hosted runners with the exception of "human interaction" steps which usually run on the normal public runners. So far no complaints to be honest.
fefe23 4y ago
This is not an experiment. This is an ad by Github who want you to buy their larger build hosts. Notably absent from the comparison is the cost for buying a build host yourself.
- yellow_lead 4y ago
  I can't even follow the math and reasoning in this post. If you're gonna do content marketing at least make it somewhat convincing. If a dev has a fixed cost for context-switching and their other task isn't a waste of time, then any build longer than a few minutes imposes that cost.
ilyt 4y ago
Back in ye olde we just linked a bunch of our machines in distcc cluster. Not really workable for remote work or really anything else but C/C++ but was pretty much all benefit for that. I'd kinda love to have transparent "workplace" that builds local by default but just have "build in cloud" option for those few cases where I do need to do a ton of compute at once (say getting to bisect something) but moving completely to the cloud seems just worse experience overall compared to just relatively beefy machine
thinkafterbef 4y ago
Disclaimer: Co-founder of https://buildjet.com/for-github-actions https://buildjet.com/for-github-actions While more cores can certainly help with certain types of projects, such as those that can be easily parallelized, this is not always the case. For example, web app projects won't benefit as much from additional cores. Another important factor to consider is the single-core performance of each vCPU. Many server-class CPUs, such as those used by GitHub, are built with a very high-core count but with a very low single-core speed. In contrast, BuildJet uses consumer CPUs, such as the 5950x, which offer slightly less core count but an excellent single-core speed. It's quite astonishing how slow "the cloud"/server-class CPUs can be, we compared my old MacBook Pro 2015 vs. a 2vCPU GitHub actions runner and the MBP 2015 won most of the time. BuildJet's bet is that single-core performance is critical for a fast CI, and it appears that the self-hosting comments here on HN also agree. (We're working our own CI, DM me if you're interested in the fastest CI on the market)
- lukeramsden 4y ago
  Super interesting, not seen BuildJet before, seems like a great concept. Sent it to a few friends that might find it useful. I wonder if there are other CI systems you could make runners for, maybe Gitlab/Buildkite? "The ultimate compute solution for your CI"
- e1g 4y ago
  > BuildJet's bet is that single-core performance is critical for a fast CI, Yes, but an even larger impact often comes from caching dependencies (node_modules/vendors/lint checks/etc). Caching via GitHub actions is slow, and if BuildJet would offer a custom GHA action with _local_ SSD caching, it would give a big advantage to people chasing fast CI.
  - thinkafterbef 4y ago
    That's precisely what we will do! You'll be able to customize the runner image however you want, and it will be running on very fast NVMe SSDs.
- tasn 4y ago
ohgodplsno 4y ago
Most people's CI issues can be solved in three easy steps. 1/ Get an actual, real machine. We run all of our CI for 10 devs out of a single 13900K / 64GB RAM machine, from a Jenkins setup that I've done over a weekend that has a single runner. 5-10 minute runs for PR validations (500kloc app, shitloads of tests, etc), 3 minutes for app builds. Buy it and host it yourself, or rent from Hetzner. You don't need a sysadmin for that. Machine is fucked ? Just restart it lol. Dependencies out of date ? Just upgrade your dockerfile, or your ansible setup and reinstall it all. Used to use Gitlab runners, Bitrise, Github Actions, etc. They're all too slow, because all these services make money on allocating as little as possible to you to stack as many clients as possible on the same server. Pretty UIs are not worth 45 minute builds. 2/ Cache. Cache. Cache. Cache. Cache. It isn't hard if you're not looking for perfect. Nobody cares if sometimes, randomly you have one small module that rebuilds even though it shouldn't have: the beefy machine is here to make sure it takes 10 seconds anyways. All that matters is that you never have a cache that says even though it shouldn't. And to solve this: don't make the cache yourself. Your build tool either has it (Gradle, Maven, Webpack, etc), or there's a widely used tool that works (ccache, etc.). How do you store your cache ? Not on the build machine. Because that fucking sucks. Make it a remote cache (If using gradle, you can pay for enterprise if you're feeling optimistic, host a gradle remote cache yourself, or use https://github.com/androidx/gcp-gradle-build-cache https://github.com/androidx/gcp-gradle-build-cache and forget about it. The running costs of it have been well under a hundred dollars over a year). 3/ Less important, but know your build tool. I've spent a lot of time rewriting Gradle projects to avoid cross configuration, allow configuration caching, allow parallel configuration, etc. The same applies to everything else. Your build tool should be part of your knowledge.
simonhamp 4y ago
I have never had to run a build that takes more than a few minutes, so I'm unfamiliar with these sorts of long-running builds. I definitely agree on investing to make builds faster, but when they're a few minutes and the build will go down to slightly fewer minutes, those 'marginal' gains don't seem worth the increased build cost. Just throwing more compute at the problem isn't always the answer either. The build tools I've been using over the past few years (Webpack and now Vite) have gotten so much faster on their own, which shows that there's loads of slack in our code and lots of room for improvement there before we need to throw more compute at it. Also, I'm certainly not just twiddling my thumbs while it's running... Am I the odd one out? Feels like a bit of a straw man argument from MS/GH trying to justify a blanket switch to more powerful runners 'because the data says so' Do the math _for your business_ and _your use-case_
- 5e92cb50239222b 4y ago
  If you're doing fullstack web dev and writing lots of end to end tests, you'll run build times into dozens of minutes without even trying (since those involve spawning web browsers…)
  - matja 4y ago
    CRIU (https://criu.org/Usage_scenarios https://criu.org/Usage_scenarios) can be used to speed-up spawning processes or duplicating state, can save significant time.
solatic 4y ago
Please, step me through this. In a start-up: a) Developer builds locally b) Developer tests locally c) Developer pushes to a repository d) CI starts e) CI ends f) Wait for human code review and approval g) Merge and deploy h) Observe that nothing broke / no need to revert. Is it even possible for (e) - (d) to be short enough, let alone (f) - (d), to keep the developer's attention instead of context switching? Most devs I know just context switch immediately after (c). If you care about developer productivity, you're much more likely to get results from focusing on (a) and (b), by hooking development laptops into caching and restricting system/test scope, than you are by reducing CI times, unless your CI takes some ungodly amount of time to run (several hours). For point of comparison: article examines how using monster machines can reduce the build time of Fedora to 27 minutes (not really a comparable example to most companies, but OK). My devs complain about an (admittedly unoptimized) CI time of 20 minutes (on a much simpler project than Fedora) that introduces context switching. Is the article really trying to get me to believe that Fedora developers wouldn't context switch on a 27 minute build, twiddling their thumbs for 27 minutes, but that a 35 or 55 minute build would? Something about getting under the 30 minute bar gets developers to keep their focus? I call bullshit.
- isoprophlex 4y ago
  Here's a single bit of anecdata: I'm currently working on a computer vision pipeline. Some pipeline components test and build in 3-5 minutes, which is the absolute upper limit on my attention. Other components take 10-20 minutes, and I always switch away to something else.
- ParetoOptimal 4y ago
  What about e.1) rerun failing CI build?
- epage 4y ago
  Agreed that there are categorical tiers of CI times (instant, check something, grab a coffee, grab a lunch, see you next day) and optimizations within a tier don't matter. I will say blocking review on CI is unneeded and, depending on your system, dramatically slows down down your whole process. PRs being posted is the main notification people get. If people instead poll for CI being done, then you'll get significant lag. This shouldn't negatively affect things. The biggest risk is CI pointing out a change big enough to invalidate the review but that has been rare in my experience. CI should just be treated as another reviewer.
jjallen 4y ago
This long waiting time for builds and testing to complete is a big part of the reason Go was created. Couldn't find a direct link. But yeah, using Go is great, build times are crazy low.
benibela 4y ago
If that was written in Pascal, it could be compiled in 25 minutes on a single core rather than using 300 minutes on a dual core
Dowwie 4y ago
This makes languages such as Rust a tough sell for startups that can't yet afford $3000 M1 laptops for each Rust developer. Rust has incremental compilation and it helps to make incremental builds faster. You also have to wait for compilation every time you run tests. Further, you wait for CI test suites to finish. Optimizing builds in CI, taking advantage of caching at the right steps, isn't widely understood nor does optimization completely address the problem with compile times. Throwing hardware at the problem makes sense, if you can afford to, but you still need to optimize the entire chain. Is a remote dev environment an economically viable alternative? I suspect not but haven't run the numbers. It's not clear to me what tier of virtual server would beat an M1 macbook pro with 32gb ram and sufficient number of cores.
lowbloodsugar 4y ago
Who is paying $15/hour for 64core 256gb??? That’s like 5x what it should be.
no_wizard 4y ago
If you're in the frontend space and are migrating to or already work in a Monorepo, I can't speak highly enough of NX[0] and their cloud builds. Very very fast. The caching algorithm they use is pretty nice and a little smarter than Turbo, and the distributed caching helps both in CI and developer machines. My only complaint is I haven't been able to quite figure out how to make the developer machine cache independent from the CI/CD one, which has caused a couple of small issues we were able to work around. One thing I have to say though, is I don't use NX plugins except for `@nrwl/jest`. I just use their Task runner capabilities. I found the plugins were a hassle if you weren't adopting the integrated[1] approach. For whatever reason, Jest worked well in the package based monorepo (for whatever reason, our IDE of choice, JetBrains, doesn't support TSConfig Paths properly in TSX files so it killed DX. So we had to go package based) That said, it was easy enough to get up and moving with the task runner and we've seen major benefits. Did require us to think a little bit differently in how to run tasks in a parallel friendly way though. [0]: https://nx.dev/ https://nx.dev/ [1]: https://nx.dev/concepts/integrated-vs-package-based https://nx.dev/concepts/integrated-vs-package-based

167 points by pimterry 4y ago

> The Bottom Line > It’s cheaper—and less frustrating for your developers—to pay more for better hardware to keep your team on track. I agree with this point, 100%. And not just in the context of build hardware. It's often cheaper to pay for specialized services rather than paying in-house developers to do those things. I've lost count of the number of times in my career that I've spent days of effort (at $200/hr) so that the company I was working for could avoid paying for some $15/mo SaaS.

deleted 4y ago
[deleted]
emerongi 4y ago
Premature optimization is the root of all evil. It's almost always better to start by using a specialized service, even if it's overpriced. Later on you can evaluate much more adequately if you'd save money/time by building your own.
- gpderetta 4y ago
  Premature "Premature optimization is the root of all evil" is the root of all evil. Recurse as needed.
EdwardDiego 4y ago
I'm experiencing the contrary, my company paying over a 100K a year for a managed service I could deploy and maintain single handed and it would still require less than 5% of my time. But you know, when you're scared of the unknown unknowns, the risk, managed services are attractive.
- ilyt 4y ago
  If we started from scratch, all the automation we have would take months to build so cloud might be cost effective for first few years. But with automation already in place cost to both maintain that and that once-a-month visit to replace a hard disk in datacenter, or once-a year half-day to install few servers is far cheaper than anything cloud, especially if you need bandwidth
- jameshart 4y ago
  The cost to a business of lost developer productivity is not the salary cost of the developer doing non dev work. It is the lost productivity of the developer not making the things you’re paying them for. The reason the business is willing to spend $x a year on the developer is because the developer produces >$x a year of value, in the form of revenue generating software. If you divert 5% of your time into doing something else, you are taking money the business wanted to invest into capital (new software features) and spending it on… something else. Lunch orders or whatever. Not making capital, anyhow.
  - kqr 4y ago
    Huh. I find myself partly disagreeing with this but I cannot for the life of me articulate why. Upvoted in the hope that a discussion is sparked...
- TYMorningCoffee 4y ago
  There also qualitative benefits like the knowledge the developer and the company gains. Also, the developer might enjoy the change of scenery.
plonk 4y ago
For us the work was mostly in adapting the old codebase to CI/CD and writing/debugging the workflows. There’s almost no additional cost to running self-hosted runners, in fact we wouldn’t be using GitHub at all if they didn’t offer a local runner.
albrewer 4y ago
My entire role at my current company could be better served by spending $60k on software licenses a year, but they choose to pay triple that for me to create a shittier version of it. I'm not complaining, really, because it's been an enormous amount of fun and I've been able to solve a ton of interesting problems, but from a business perspective it makes absolutely no sense.

tjpnz 4y ago

Are there really people who still need convincing of this? At the same time you probably shouldn't be throwing more hardware at the problem without trying to understand it first. I've witnessed numerous times throughout my career what can be saved by tuning a few parameters.

papito 4y ago
It's the resource curse. Why think about it too much if you can just throw resources at the problem? This is not just an issue with builds. Seems like developers treat everything as "free", from HTTP round-trips to database queries. Not to mention the tendency to craft convoluted and heavy solutions to rather standard problems. Because "this is how it's done now". Standing here, holding my hammer. Lo, on the horizon: a nail!
KronisLV 4y ago
> Are there really people who still need convincing of this? Some might just not care that much, or nobody might speak up about the problem, not wanting to make it seem like they're just complaining, in the case of nobody being able to provision more resources, or optimize the build. For example, in some environments, getting a better CI node might be associated with red tape, lots of back and forth with a different department or something like that, especially if it's all on-prem instead of using managed services or cloud resources. > At the same time you probably shouldn't be throwing more hardware at the problem without trying to understand it first. I've witnessed numerous times throughout my career what can be saved by tuning a few parameters. This is an excellent point! In a context of containers (just an example, though illustrates some pitfalls nicely) there's a world of difference between using a base image that has the runtime and tools you need, vs installing them every time. There's a lot of difference between pulling in your dependencies every time, vs using a cache for it or an instance of Nexus that acts as a caching proxy (though that's not very nice to configure in most languages). There's even difference between making sure that packages install correctly from the cache, vs having them already be installed in some intermediate layer that you can just reuse when building your container. Even without containers, there can be lots of things to take into consideration, like whether you could build your back end and front end (in the context of webdev) or other components in parallel, whether you could run any tests you might have in parallel, or whether there's not duplicated work going on in there somewhere.

carreau 4y ago

I my experience if you get more builders/faster hardware someone higher up will end up asking for more items in the build-matrix, and the CI time will balloon up again. It's rarely only a question of just allocating money to more/better hardware, it's also a question of policy and willingness of your organisation to keep CI time short/feedback fast.

substation13 4y ago
The solution is a build system that scales by caching between runs - most PRs only change a few files, after all.
- carreau 4y ago
  As I replied to the sibling comment, yes, I agree, but caching is not magic. You need to know what to invalidate, and you need to be allowed to spend time making test group that run depending on what have changed. So without policy and having devs being allowed to work on optimizing CI/reducing time, you can list plenty of strategies to make it faster but it will likely not happen.
  - ohgodplsno 4y ago
    Caching is pretty damn simple with any sensible build tool that simply stores hashes of files and modules. Caching in the JVM ecosystem is pretty much free, for example.
    - carreau 4y ago
      Sure, but you seem to be read this blog post in the narrow view of building as only "compiling an artifact" and not running the test suite across many platforms/condition, or any other work. I mean the author even points out that building the kernel is only a use-case that takes time for measurement purposes. For some projects I work on the compilation itself takes maybe 1/10 max of the total build and test. So even with infinite agressive compilation caching I would not gain much.

Hardwired8976 4y ago

I worked from home on Wednesday and I managed to watch one episode of Better Call Saul while waiting for the compilation of my code lol.

titzer 4y ago
Chrome dev?
- endgame 4y ago
  Nah, Chrome is larger than that. I built NixOS 22.11 from source and Chromium took over 6 hours on a Ryzen 7 3800XT with 32GB of RAM. Had to run the build on spinning rust though, because neither my SSD and certainly not my tmpfs were large enough.
- Hardwired8976 4y ago
  Cobol, the crazy thing is I gotta wait the 5min before I can even debug it as we aren't allowed to run it locally
  - pharmakom 4y ago
    Someone should pause on the Better Call Saul and write some Bazel rules for COBOL.

tjoff 4y ago

This is also a big reason to self-host. Without this beta github offer: 2-core CPU (x86_64), 7 GB of RAM, 14 GB of SSD space Your last development machine that was thrown away is so much faster than this (and we do have great tools for administer stuff like this nowadays). Hell, the computer I bought as a student in 2008 is comparable(!) (though it didn't have an SSD). And it will have so much better network connectivity with everything else on your network. Whenever you hear cloud, realize that the dedicated time you will get (unless you specifically pay for it (in which case it will be more expensive than self-host)) is in the same order the phone you had two generations ago. That is why gmail search sucks. Because they can't afford to really search your messages and can not even do exact matching properly. So yes, apparently github is fixing this now and if paying for this problem makes sense for you do it. But this is a problem that was partly invented by the cloud in the first place.

qersist3nce 4y ago
>Your last development machine that was thrown away is so much faster than this >the computer I bought as a student in 2008 is comparable >the phone you had two generations ago Good for Americans (or first-world citizens)
- danuker 4y ago
  As a Romanian web dev, my 8-y-o laptop has a 4-core CPU, 12GB RAM, and 256GB of SSD space. The laptop cost about 110% of my monthly entry-level salary.
  - 5e92cb50239222b 4y ago
    Romania is significantly better developed and more affluent than most of the world. My similarly aged laptop had much worse hardware than what you're describing and cost 200%+ of my old entry-level salary. (I don't use laptops anymore.) GitHub CI also has much better network connectivity (both speed and especially ping) compared to most parts of my country.
- ddorian43 4y ago

rpigab 4y ago

That's why you should only hire contractors, and shouldn't pay them while a build is running. Saves more money that way, and you don't have to care much about build duration.

panzerboiler 4y ago
Hallo. I am a contractor. Why you shouldn't pay for the time my machine is building your project?
- wruza 4y ago
  Cause you can work on other projects meanwhile, multitasking is easy and never helps making mistakes. It’s also easy to contract on your terms. You just say “I want that for very cheap” and get flooded with interested contractors. /s
DocTomoe 4y ago
Good luck finding contractors who will agree to these terms. Waiting for built times is dead time which is filled with other activity, like documentation, or reading through the next requirement. What you will achieve with this policy is minimizing the number of build processes, from a few times a day to once or twice a week (or less), which is exactly what you do not want from a CI/CD / DevOps environment.
- randomdata 4y ago
  Besides, you're not paying for one's time. You're paying for one's availability. The difference is subtle, but significant.
chronial 4y ago
Is this sarcasm about our absurd reality of hyper-capitalism or are you serious?
- rpigab 4y ago
  I'm dead serious, we have to embrace the gig economy and everything it has to change in our lives for the better. Why would I be paying for bathroom breaks and idle time? I wish there was software to count how much time is lost when users minimize all windows or switch to desktop, so I could deduct it from their pay. I know some of this idle time is spent thinking about problems to solve related to the job, but it's something that can be done in your free time, or while commuting.

iLoveOncall 4y ago

> In short, if a developer is waiting on a build to run for one hour and doing nothing in that timeframe This should be at the very top of the article. It basically negates entirely the whole experiment because no developer will just sit and wait for hours for a build (unless they planned on taking that break anyway). The cost is not 0 but it is absolutely negligible.

bombolo 4y ago
https://xkcd.com/303/ https://xkcd.com/303/
Aeolun 4y ago
I don’t know what work you do, but if my test takes longer than 30 minutes I’ll start working on something else, and when I come back to fix whatever failed in the test (when I get to it a few hours later), I’ll have to reconstruct the entire problem space in my head again.
- iLoveOncall 4y ago
  If your build time or test runtime takes 30 minutes I'd argue your problem won't be solved by faster hardware.
tener 4y ago
This requires a context switch which is expensive in it's own way. Read the article, it covers this pattern pretty well.
- bstpierre 4y ago
  Even with the high end machine it still takes 27 minutes. If I have a 27 minute build I am absolutely going to context switch. If it’s more than about 3 minutes I am probably going to context switch. An interesting experiment would be to find the switch-time threshold for a wide range of developers. Typically I don’t rely on a full build for getting things accomplished, and not CI if I can help it. Local, incremental, and minimal path to feedback is what I aim for.

e1g 4y ago

I recently moved our GutHub CI to a self-hosted runner and it reduced CI times for all jobs by ~5x. One frustrating part is that a single GitHub runner can handle one job at a time, and our repos have 4-8 highly parallelized jobs. So we need multiple runners. To do this in an economic way, I made a Docker image and run 10 instances of it a single dedicated host from Hetzner. For ~$50/mo, we have unlimited CI and job run as fast as theoretically possible. The CI box has no inbound ports open to the internet, and the OS is set to auto-update, so the maintenance burden is low.

Delemono 4y ago
That sounds more like expected than an issue. You might not want to overload one instance. And running it like you do, is a no brainer anyway
- e1g 4y ago
  Executing a GitHub runner in a container is a no-brainer, but it's still easy to overload the _host_ with too many jobs/containers, so managing resources is always up to the operator. I understand why GitHub has this constraint - to avoid clashes between jobs. If multiple jobs (from the same repository) are executing simultaneously within the same stateful environment, they are more likely to clash over shared resources (/tmp, cleanup tasks, database names, etc.). However, even if my jobs are clean and idempotent, GitHub is nudging me to think about runners as "VMs" rather than "containers" (because often CI jobs involve their own containers, and docker-in-docker is a pain), and "self-hosting a bunch of CI VMs" becomes expensive rather quick.
  - naikrovek 4y ago
    well, GHA supports running actions in containers, which is the best way to control the environment your CI/CD runs in, so putting your runner in a container won't work for everyone. but, if it does work for someone, doing what you've done will give a much better experience to their developers. my employer uses single-user VMs for it's runners; it works well, but sometimes actions invocation is high and it can take a few minutes for a runner to come around to taking my job. that would be much less of a problem with dockerized runners.

malinens 4y ago

One of the main metric for build time for me is to make it fast enough so you do not want switch to other task at all

Cthulhu_ 4y ago
Same, but this is why I will never even consider waiting for a CI pipeline; even if it's fast, there's always overhead in waiting for the pipeline to start, setting up the runners, cloning git, transferring files, etc. I always see it as a hierarchy; editor, local runtime, pre-commit / pre-push hooks, CI. At the moment, if this setup works, my pre-push hook should only run various tools (linters, unit tests) on the files that were affected by the commit. I'll leave it to CI to run the full suite of verifications and end-to-end tests.
alkonaut 4y ago
Devs need quick local builds to not lose their train of thought. But even when a build takes 5 seconds, a test suite might take 5 hours. CI is usually dominated by tests (and other non-compilation tasks such as packaging, static analysis, obfuscation, signing, even). Local builds can often be incremental and finish in seconds even if a full build takes ten minutes. My CI is basically 5% git checkout (varies), 2% build (a few minutes), 80% run tests, 5% static analysis, the rest is misc crap like uploads of artifacts, code signing, etc.

marginalia_nu 4y ago

Excepting C++ development, is slow build times really a hardware problem? CI software in general is very slow. But even building locally, there are many projects that have just ridiculous build times because it's just not a priority at all. I've seen small-medium sized Java projects with a 25 minute test suite. This has been in part due to extremely inefficient tests that keep assembling the entire system over and over each test, often with extreme redundancy in coverage, ensuring you need to keep running the entire suite whenever a test breaks. Nobody is going to run tests for half an hour unless you hold a gun to their head. So you make your change by 10 AM, usually something trivial, and then you spend the afternoon fixing the 30 tests that broke, running the suite over and over to find more broken tests to fix. It's almost comically inefficient.

Aeolun 4y ago
I mean, you spending the afternoon fixing all the tests that broke is a lot more efficient than you spending the night fixing those same issues when it gets deployed to production.
- marginalia_nu 4y ago
  Sure, but I'm not convinced this necessary or sufficient to prevent that. It's rarely obvious what these types of tests actually test. It's like... just a bit of everything for the sake of good coverage. That and the inertia from having to wait half an hour for the test results makes it very compelling to start just "fixing" the tests by changing the assertions to expect the new value. If you have a more sensible testing strategy, it's obvious what is being tested (and thus also relatively clear what broke), and the turnaround is seconds possibly low single-digit minutes, so you don't have to rebuild the mental context every time you run the tests.
  - newaccount74 4y ago
    > "fixing" the tests by changing the assertions to the new value Oh my... I spent hours trying to figure out why I broke a test, only to figure out that my change actually fixed incorrect behavior! The previous dev who worked on the code had "fixed" the test failure by changing the assertion to expect the incorrect behavior! Automatic test suites with thousands of tests sound great in theory, but in practice it's not all rosy...

kunley 4y ago

Hmm, in addition to that, what about un-bloating the typical CICD to speed up the build?

Aeolun 4y ago
What part would you like to unbloat? Github is pretty light as far as I’m concerned.

tikkabhuna 4y ago

We use GitLab CI and solve this with tags. We have runners allocated with set resources and tagged with a t-shirt size. Developers can then tag their job based on this size. Single threaded jobs (curl-uploading a file) can be tagged "small" to get 1 CPU core. Building projects can be medium or large based on how big they are and developers can choose whether the large runner is needed. Because its simple to switch tags, developers can create a feature branch to swap the tag and observe how long it takes. All our runners are self hosted. There are fewer large runners, so developers are inclined to go small. Hypothetically, you can run all your jobs in large but then you may have to wait for availability but in practice we haven't seen any real contention. We run over 15,000 jobs a week on 6 physical servers with this pattern.

deleted 4y ago

[deleted]

pron 4y ago

Focusing on the cost side of software development is not very convincing for managers who need to make purchasing decisions. I believe most software is a high-margin product because manufacturing costs are close to zero. Even a relatively high difference in cost is not as much as a small difference in value. The one cost-side metric that has a big effect on value is time-to-market. What could be more convincing is the effect on product value that build times could have, and they certainly could (and I, for one, believe that they do, especially if they can be made to be less than 15 seconds or so). For example, it's certainly conceivable that very short build times make it easier to write more tests, which could result in more correct software with better features and/or a shorter time-to-market. Some products targeted at developers also make the mistake of focusing their marketing message on the cost, perhaps because that's the part of the business that developers personally feel, but it's not the message that would convince their employers.

DrScientist 4y ago
Totally agree. What matters is development velocity and quality - these are the things you will be competing on - not really cost of production. Slow builds impact both of the above of course.
chasinglogic 4y ago
Definitely, it's also important to remember how salaries vs. infrastructure costs show up on a P&L sheet. Salaries are going to be considered largely immovable and they won't "go down" or show as a lower number on the P&L as a result of your devs getting some time back. Whereas your infrastructure costs jumping up (even by a small amount, esp if you have any amount of scale) will set of alarms, rightly IMO, in the Finance department.
- pron 4y ago
  Yep. In developer-targeted products the users are usually not the customers. Making the product appealing to users is (often) necessary, but it's insufficient to convince the actual customer. Sometimes the problem is in the product itself, but often it's just a matter of shaping the marketing message.

sluongng 4y ago

This is also a reason why many large organizations recently started to adopt a matured build tool solution such as: Bazel / Pants / Buck. They did the math.

moffkalast 4y ago
But did they also do the monster math?
- EdwardDiego 4y ago
  It was a Bazel smash.

alkonaut 4y ago

Build is one of those things that I can't see the economy in buying cloud VMs for if you have lots of load. Buy a rack and a dozen cheap-ish rack chassis with some middle range CPUs with lots of cores but not necessarily the highest specs otherwise (Last gen we used 9700K and for a recent refresh it's AMD 7700X). You can get quite far at $1k per node and $12k for a rack. Then you can run 3-4 build agents per node so you have a 48 agent cluster in your rack. Electricity and management will be on top of the purchase price of course, but it's still a bargain. Even if you do use cloud compute for builds, it's worth having some self-hosted nodes for the base need and just using the cloud for peak use scaling.

hhh 4y ago
The cost of someone to manage and support this in my org would cost more than it is worth. Gitlab runners on an autoscaling group handles load for me and scales up and down with need. Took 2-3 hours to set up, and has lasted 2 years without incident. The value for me is cost (especially in time) to implement, mental burden, and risk of delaying projects.
- alkonaut 4y ago
  It depends on what resources you have on hand obviously, but perhaps more it will depend on resource usage. There is little point buying a rack and having it idle. If you have very bursty workloads then scaling pays off. If you run your rick nice and warm even over nights and weekends then it's more economic to have self hosted I imagine.

ttyyzz 4y ago

As a Developer, build times sometimes are a blessing :D But thats beside the point. In my JavaScript frontend Framework / Nodejs backend world, build times are often times horrific due to the Single-threaded nature of JS. Hardware is often times not to blame.

M4v3R 4y ago
It really depends on which bundler/framework you use. We're developing a medium-sized Next.js 12 app recently and the build times are pretty good, the production build on my machine takes around 40 seconds.
moffkalast 4y ago
Gotta love getting paid for sitting around doing nothing :P
Cthulhu_ 4y ago
That's rapidly changing thankfully with newer tooling like Deno, Vite, etc; tools that are written in a native language (rust) instead of JS. I don't think it's to blame on the single-threadedness though, I've found that i/o is more often a bottleneck, thanks to the Node ecosystem's focus on many small libraries. I've seen improved performance in that regard in Yarn 2/3 though, since it stores dependencies in .zip files that are kinder on most storage mediums and it keeps a central repository instead of duplicating dependencies across projects.

avodonosov 4y ago

If you want to speedup build of a large Maven project, check https://github.com/avodonosov/hashver-maven-plugin https://github.com/avodonosov/hashver-maven-plugin

jurmous 4y ago

[deleted by author]

skrebbel 4y ago
Please stop
- jurmous 4y ago
  Ok. Was the first time I tried this on a post and found it an interesting take so wanted to share but wanted to be transparent it was AI generated. But maybe others are doing it too much already which I am not aware of yet.

nottorp 4y ago

Isn't the real problem that no one does partial local builds any more? And maybe the lack of a local testing environment. Build servers are fine for the main branch, but why does a dev have to build the whole thing to quickly check the last 50 lines that they've written?

amtamt 4y ago
So many times it is not even 50, but 1 line. and those changes fail too. Also, more hardware does not make builds faster magically all the times. one needs to spend time to actually use more/ better hardware. Finally, build load comes in spikes. end of the working day, you have lots of builds all of a sudden. Few runners get overloaded in peak or many runners waste a lot of money. May be we need builds being done in some other timezones where its off-peak time, and even better, if people could pool their resources (and thats what SaaS runners do for us.)
treis 4y ago
>Isn't the real problem that no one does partial local builds any more? And maybe the lack of a local testing environment. I die a little inside every time my build is waiting for a runner while my M1 Mac is just sitting there.
- mdaniel 4y ago
  AFAIK there's nothing stopping you from just spinning one up if you have the perms to the repo, although because of both your M1 and "untrusted code" reasons you'd likely want to put it in a virtual machine: https://docs.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository https://docs.github.com/en/actions/hosting-your-own-runners/... GitLab behaves similarly
chillfox 4y ago
Because we decided for some reason that CI builds (the build config) should not be able to be run locally for testing.
bluGill 4y ago

comprev 4y ago

We use Github Actions on self-hosted runners with the exception of "human interaction" steps which usually run on the normal public runners. So far no complaints to be honest.

fefe23 4y ago

This is not an experiment. This is an ad by Github who want you to buy their larger build hosts. Notably absent from the comparison is the cost for buying a build host yourself.

yellow_lead 4y ago
I can't even follow the math and reasoning in this post. If you're gonna do content marketing at least make it somewhat convincing. If a dev has a fixed cost for context-switching and their other task isn't a waste of time, then any build longer than a few minutes imposes that cost.

ilyt 4y ago

Back in ye olde we just linked a bunch of our machines in distcc cluster. Not really workable for remote work or really anything else but C/C++ but was pretty much all benefit for that. I'd kinda love to have transparent "workplace" that builds local by default but just have "build in cloud" option for those few cases where I do need to do a ton of compute at once (say getting to bisect something) but moving completely to the cloud seems just worse experience overall compared to just relatively beefy machine

thinkafterbef 4y ago

Disclaimer: Co-founder of https://buildjet.com/for-github-actions https://buildjet.com/for-github-actions While more cores can certainly help with certain types of projects, such as those that can be easily parallelized, this is not always the case. For example, web app projects won't benefit as much from additional cores. Another important factor to consider is the single-core performance of each vCPU. Many server-class CPUs, such as those used by GitHub, are built with a very high-core count but with a very low single-core speed. In contrast, BuildJet uses consumer CPUs, such as the 5950x, which offer slightly less core count but an excellent single-core speed. It's quite astonishing how slow "the cloud"/server-class CPUs can be, we compared my old MacBook Pro 2015 vs. a 2vCPU GitHub actions runner and the MBP 2015 won most of the time. BuildJet's bet is that single-core performance is critical for a fast CI, and it appears that the self-hosting comments here on HN also agree. (We're working our own CI, DM me if you're interested in the fastest CI on the market)

lukeramsden 4y ago
Super interesting, not seen BuildJet before, seems like a great concept. Sent it to a few friends that might find it useful. I wonder if there are other CI systems you could make runners for, maybe Gitlab/Buildkite? "The ultimate compute solution for your CI"
e1g 4y ago
> BuildJet's bet is that single-core performance is critical for a fast CI, Yes, but an even larger impact often comes from caching dependencies (node_modules/vendors/lint checks/etc). Caching via GitHub actions is slow, and if BuildJet would offer a custom GHA action with _local_ SSD caching, it would give a big advantage to people chasing fast CI.
- thinkafterbef 4y ago
  That's precisely what we will do! You'll be able to customize the runner image however you want, and it will be running on very fast NVMe SSDs.
tasn 4y ago

ohgodplsno 4y ago

Most people's CI issues can be solved in three easy steps. 1/ Get an actual, real machine. We run all of our CI for 10 devs out of a single 13900K / 64GB RAM machine, from a Jenkins setup that I've done over a weekend that has a single runner. 5-10 minute runs for PR validations (500kloc app, shitloads of tests, etc), 3 minutes for app builds. Buy it and host it yourself, or rent from Hetzner. You don't need a sysadmin for that. Machine is fucked ? Just restart it lol. Dependencies out of date ? Just upgrade your dockerfile, or your ansible setup and reinstall it all. Used to use Gitlab runners, Bitrise, Github Actions, etc. They're all too slow, because all these services make money on allocating as little as possible to you to stack as many clients as possible on the same server. Pretty UIs are not worth 45 minute builds. 2/ Cache. Cache. Cache. Cache. Cache. It isn't hard if you're not looking for perfect. Nobody cares if sometimes, randomly you have one small module that rebuilds even though it shouldn't have: the beefy machine is here to make sure it takes 10 seconds anyways. All that matters is that you never have a cache that says even though it shouldn't. And to solve this: don't make the cache yourself. Your build tool either has it (Gradle, Maven, Webpack, etc), or there's a widely used tool that works (ccache, etc.). How do you store your cache ? Not on the build machine. Because that fucking sucks. Make it a remote cache (If using gradle, you can pay for enterprise if you're feeling optimistic, host a gradle remote cache yourself, or use https://github.com/androidx/gcp-gradle-build-cache https://github.com/androidx/gcp-gradle-build-cache and forget about it. The running costs of it have been well under a hundred dollars over a year). 3/ Less important, but know your build tool. I've spent a lot of time rewriting Gradle projects to avoid cross configuration, allow configuration caching, allow parallel configuration, etc. The same applies to everything else. Your build tool should be part of your knowledge.

simonhamp 4y ago

I have never had to run a build that takes more than a few minutes, so I'm unfamiliar with these sorts of long-running builds. I definitely agree on investing to make builds faster, but when they're a few minutes and the build will go down to slightly fewer minutes, those 'marginal' gains don't seem worth the increased build cost. Just throwing more compute at the problem isn't always the answer either. The build tools I've been using over the past few years (Webpack and now Vite) have gotten so much faster on their own, which shows that there's loads of slack in our code and lots of room for improvement there before we need to throw more compute at it. Also, I'm certainly not just twiddling my thumbs while it's running... Am I the odd one out? Feels like a bit of a straw man argument from MS/GH trying to justify a blanket switch to more powerful runners 'because the data says so' Do the math _for your business_ and _your use-case_

5e92cb50239222b 4y ago
If you're doing fullstack web dev and writing lots of end to end tests, you'll run build times into dozens of minutes without even trying (since those involve spawning web browsers…)
- matja 4y ago
  CRIU (https://criu.org/Usage_scenarios https://criu.org/Usage_scenarios) can be used to speed-up spawning processes or duplicating state, can save significant time.

solatic 4y ago

Please, step me through this. In a start-up: a) Developer builds locally b) Developer tests locally c) Developer pushes to a repository d) CI starts e) CI ends f) Wait for human code review and approval g) Merge and deploy h) Observe that nothing broke / no need to revert. Is it even possible for (e) - (d) to be short enough, let alone (f) - (d), to keep the developer's attention instead of context switching? Most devs I know just context switch immediately after (c). If you care about developer productivity, you're much more likely to get results from focusing on (a) and (b), by hooking development laptops into caching and restricting system/test scope, than you are by reducing CI times, unless your CI takes some ungodly amount of time to run (several hours). For point of comparison: article examines how using monster machines can reduce the build time of Fedora to 27 minutes (not really a comparable example to most companies, but OK). My devs complain about an (admittedly unoptimized) CI time of 20 minutes (on a much simpler project than Fedora) that introduces context switching. Is the article really trying to get me to believe that Fedora developers wouldn't context switch on a 27 minute build, twiddling their thumbs for 27 minutes, but that a 35 or 55 minute build would? Something about getting under the 30 minute bar gets developers to keep their focus? I call bullshit.

isoprophlex 4y ago
Here's a single bit of anecdata: I'm currently working on a computer vision pipeline. Some pipeline components test and build in 3-5 minutes, which is the absolute upper limit on my attention. Other components take 10-20 minutes, and I always switch away to something else.
ParetoOptimal 4y ago
What about e.1) rerun failing CI build?
epage 4y ago
Agreed that there are categorical tiers of CI times (instant, check something, grab a coffee, grab a lunch, see you next day) and optimizations within a tier don't matter. I will say blocking review on CI is unneeded and, depending on your system, dramatically slows down down your whole process. PRs being posted is the main notification people get. If people instead poll for CI being done, then you'll get significant lag. This shouldn't negatively affect things. The biggest risk is CI pointing out a change big enough to invalidate the review but that has been rare in my experience. CI should just be treated as another reviewer.

jjallen 4y ago

This long waiting time for builds and testing to complete is a big part of the reason Go was created. Couldn't find a direct link. But yeah, using Go is great, build times are crazy low.

benibela 4y ago

If that was written in Pascal, it could be compiled in 25 minutes on a single core rather than using 300 minutes on a dual core

Dowwie 4y ago

This makes languages such as Rust a tough sell for startups that can't yet afford $3000 M1 laptops for each Rust developer. Rust has incremental compilation and it helps to make incremental builds faster. You also have to wait for compilation every time you run tests. Further, you wait for CI test suites to finish. Optimizing builds in CI, taking advantage of caching at the right steps, isn't widely understood nor does optimization completely address the problem with compile times. Throwing hardware at the problem makes sense, if you can afford to, but you still need to optimize the entire chain. Is a remote dev environment an economically viable alternative? I suspect not but haven't run the numbers. It's not clear to me what tier of virtual server would beat an M1 macbook pro with 32gb ram and sufficient number of cores.

lowbloodsugar 4y ago

Who is paying $15/hour for 64core 256gb??? That’s like 5x what it should be.

no_wizard 4y ago

If you're in the frontend space and are migrating to or already work in a Monorepo, I can't speak highly enough of NX[0] and their cloud builds. Very very fast. The caching algorithm they use is pretty nice and a little smarter than Turbo, and the distributed caching helps both in CI and developer machines. My only complaint is I haven't been able to quite figure out how to make the developer machine cache independent from the CI/CD one, which has caused a couple of small issues we were able to work around. One thing I have to say though, is I don't use NX plugins except for `@nrwl/jest`. I just use their Task runner capabilities. I found the plugins were a hassle if you weren't adopting the integrated[1] approach. For whatever reason, Jest worked well in the package based monorepo (for whatever reason, our IDE of choice, JetBrains, doesn't support TSConfig Paths properly in TSX files so it killed DX. So we had to go package based) That said, it was easy enough to get up and moving with the task runner and we've seen major benefits. Did require us to think a little bit differently in how to run tasks in a parallel friendly way though. [0]: https://nx.dev/ https://nx.dev/ [1]: https://nx.dev/concepts/integrated-vs-package-based https://nx.dev/concepts/integrated-vs-package-based