3 ms·

We've found that distributed building has pretty much eliminated the need to upgrading developer workstations. Super easy to set up, too.

packetlost 3y ago
Distributed building of what? Because for every language the answer of whether it's easy or not is probably different.
- LanzVonL 3y ago
  [flagged]
  - packetlost 3y ago
    Sooooo... C? You didn't answer the question and I would be surprised to hear that freaking Java had support for distributed compilation, and that has been around since the 90s. Take your pretentious non-answers elsewhere please.
    - LanzVonL 3y ago
      [flagged]
      - deleted 3y ago
        [deleted]
lawrjone 3y ago
I’m not sure this would work well for our use case. The distributed build systems only really benefit from aggressively caching the modules that are built, right? But the majority of the builds we do are almost fully cached, having changed just one module that needs recompiling then the linker sticks everything back together, which the machines would then need to download from the distributed builder and at 300MB a binary that’s gonna take a while. I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine? I suspect we wouldn’t want this anyway (lots of our company work on the go, train WiFi wouldn’t cut it for this!) but interested nonetheless.
- dist-epoch 3y ago
  > The distributed build systems only really benefit from aggressively caching the modules that are built, right Not really, you have more cores to build on. Significant difference for slow to compile languages like C++. > I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine? Yes, again, for C++.
closeparen 3y ago
A Macbook-equivalent AWS instance prices to at least the cost of a Macbook per year.
- lawrjone 3y ago
  Yes I actually did the maths on this. If you want a GCP instance that is comparable to an M3 Pro 36GB, you're looking at an n2-standard-8 with a 1TB SSD, which comes out at $400/month. Assuming you have it running just 8 hours a day (if your developers clock in at exact times) and you can 1/3 that to make it $133/month, or $1600/year. We expect these MacBooks to have at least a 2 year life, which means you're comparing the cost of the MacBook to 2 years of running the VM for 8 hours a day, which means $2800 vs $3200, so the MacBook still comes in $400 cheaper over it's lifetime. And the kicker is you still need to buy people laptops so they can connect to the build machine, and you can no longer work if you have bad internet connection. So for us the trade-off doesn't work whichever way you cut it.
  - throwaway892238 3y ago
    1. With a savings plan or on-demand? 2. Keeping one instance on per developer indefinitely, or only when needed? 3. Shared nodes? Node pools? 4. Compared to what other instance types/sizes? 5. Spot pricing? Shared nodes brought up on-demand with a savings plan and spot pricing is the same cost if not cheaper than dedicated high-end laptops. And on top of that, they can actually scale their resources much higher than a laptop can, and do distributed compute/test/etc, and match production. And with a remote dev environment, you can easily fix issues with onboarding where different people end up with different setups, miss steps, need their tooling re-installed or to match versions, etc.
    - lawrjone 3y ago
      1. That was assuming 8 hours of regular usage a day that has GCP's sustained use discounts applied, though not the committed usage discounts you can negotiate (but this is hard if you don't want 24/7 usage). 2. The issue with only-when-needed is the cold-start time starts hurting you in ways we're trying to pay to avoid (we want <30s feedback loops if possible) as would putting several developers on the same machine. 3. Shared as in cloud multi-tenant? Sure, we wouldn't be buying the exclusive rack for this. 4. n2-standard-8 felt comparable. 5. Not considered. If it's interesting, we run a build machine for when developers push their code into a PR and we build a binary/container as a deployable artifact. We have one machine running a c3-highcpu-22 which is 22 CPUs and 44GB memory. Even at the lower frequency of pushes to master the build latency spikes a lot on this machine when developers push separate builds simultaneously, so I'd expect we'd need a fair bit more capacity in a distributed build system to make the local builds (probably 5-10x as frequent) behave nicely.
boricj 3y ago
At one of my former jobs, some members of our dev team (myself included) had manager-spec laptops. They were just good enough to develop and run the product on, but fairly anemic overall. While I had no power over changing the laptops, I was co-administrator of the dev datacenter located 20 meters away and we had our own budget for it. Long story short, that dev datacenter soon had a new, very beefy server dedicated for CI jobs "and extras". One of said extras was providing Docker containers to the team for running the product during development, which also happened to be perfectly suitable for remote development.

packetlost 3y ago

Distributed building of what? Because for every language the answer of whether it's easy or not is probably different.

LanzVonL 3y ago
[flagged]
- packetlost 3y ago
  Sooooo... C? You didn't answer the question and I would be surprised to hear that freaking Java had support for distributed compilation, and that has been around since the 90s. Take your pretentious non-answers elsewhere please.
  - LanzVonL 3y ago
    [flagged]
    - deleted 3y ago
      [deleted]

lawrjone 3y ago

I’m not sure this would work well for our use case. The distributed build systems only really benefit from aggressively caching the modules that are built, right? But the majority of the builds we do are almost fully cached, having changed just one module that needs recompiling then the linker sticks everything back together, which the machines would then need to download from the distributed builder and at 300MB a binary that’s gonna take a while. I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine? I suspect we wouldn’t want this anyway (lots of our company work on the go, train WiFi wouldn’t cut it for this!) but interested nonetheless.

dist-epoch 3y ago
> The distributed build systems only really benefit from aggressively caching the modules that are built, right Not really, you have more cores to build on. Significant difference for slow to compile languages like C++. > I may have this totally wrong though. Would distributed builds actually get us a new binary faster to the local machine? Yes, again, for C++.

closeparen 3y ago

A Macbook-equivalent AWS instance prices to at least the cost of a Macbook per year.

lawrjone 3y ago
Yes I actually did the maths on this. If you want a GCP instance that is comparable to an M3 Pro 36GB, you're looking at an n2-standard-8 with a 1TB SSD, which comes out at $400/month. Assuming you have it running just 8 hours a day (if your developers clock in at exact times) and you can 1/3 that to make it $133/month, or $1600/year. We expect these MacBooks to have at least a 2 year life, which means you're comparing the cost of the MacBook to 2 years of running the VM for 8 hours a day, which means $2800 vs $3200, so the MacBook still comes in $400 cheaper over it's lifetime. And the kicker is you still need to buy people laptops so they can connect to the build machine, and you can no longer work if you have bad internet connection. So for us the trade-off doesn't work whichever way you cut it.
- throwaway892238 3y ago
  1. With a savings plan or on-demand? 2. Keeping one instance on per developer indefinitely, or only when needed? 3. Shared nodes? Node pools? 4. Compared to what other instance types/sizes? 5. Spot pricing? Shared nodes brought up on-demand with a savings plan and spot pricing is the same cost if not cheaper than dedicated high-end laptops. And on top of that, they can actually scale their resources much higher than a laptop can, and do distributed compute/test/etc, and match production. And with a remote dev environment, you can easily fix issues with onboarding where different people end up with different setups, miss steps, need their tooling re-installed or to match versions, etc.
  - lawrjone 3y ago
    1. That was assuming 8 hours of regular usage a day that has GCP's sustained use discounts applied, though not the committed usage discounts you can negotiate (but this is hard if you don't want 24/7 usage). 2. The issue with only-when-needed is the cold-start time starts hurting you in ways we're trying to pay to avoid (we want <30s feedback loops if possible) as would putting several developers on the same machine. 3. Shared as in cloud multi-tenant? Sure, we wouldn't be buying the exclusive rack for this. 4. n2-standard-8 felt comparable. 5. Not considered. If it's interesting, we run a build machine for when developers push their code into a PR and we build a binary/container as a deployable artifact. We have one machine running a c3-highcpu-22 which is 22 CPUs and 44GB memory. Even at the lower frequency of pushes to master the build latency spikes a lot on this machine when developers push separate builds simultaneously, so I'd expect we'd need a fair bit more capacity in a distributed build system to make the local builds (probably 5-10x as frequent) behave nicely.

boricj 3y ago

At one of my former jobs, some members of our dev team (myself included) had manager-spec laptops. They were just good enough to develop and run the product on, but fairly anemic overall. While I had no power over changing the laptops, I was co-administrator of the dev datacenter located 20 meters away and we had our own budget for it. Long story short, that dev datacenter soon had a new, very beefy server dedicated for CI jobs "and extras". One of said extras was providing Docker containers to the team for running the product during development, which also happened to be perfectly suitable for remote development.