4 ms·

At my work senior leads and architects always shoot down being able to develop locally, especially when devs bring up wanting the capability. The critique is so

At my work senior leads and architects always shoot down being able to develop locally, especially when devs bring up wanting the capability. The critique is some variation of “it would be impossible to run all services locally” or “it would be too expensive.” So we develop the code and deploy into QA, often breaking things. Cycle times are measured in hours to days. Breaking QA is sometimes a fireable offense. Lol. The leads and architects are correct in my case; it would be impossible and too expensive to do. This is because our services are built on hundreds of bespoke manual configurations made along the way to production. Discovering and pulling into code/automation would be a whole months/year long project in itself. That said there are ways of developing locally without running everything locally. Pull in the code of the service you want to work on locally and just point to QA for its dependencies. Most times it takes some finagling of code but it usually works. Even if everything was running locally, often generating usable data is the biggest barrier. In QA thousands of hands have generated partly useable data. It’s a hard problem to solve since I don’t want to have to know about data requirements for every service.

rcxdude 2y ago
Yeah, this is the situation being argued against. If you don't focus on making your environment reproducible, you wind up tangled in a mess no-one can understand. I imagine you also don't expect to be able to get things running again after a disaster strikes.
tetraodonpuffer 2y ago
I don’t understand some of my colleagues that feel this way either, I’ve been around a long time but having a way to set up a ”local” environment with a local automated build to me seems priority one for any piece of software cloud or otherwise. By “local” I mean self contained, my “local” environment is right now a single cloud Linux VM with kind for kubernetes, all sorts of k8s local monitoring / cert issuing / stubbed things, metallb and openvpn to access (with a local to my computer dhcpcd for DNS resolution) and so on, all building from local and orchestrated by a python set of scripts. All of this works great and has saved me so many times and made development so much faster compared to teams that rely on ci/cd even for basic development (where every commit takes an hour+ to deploy due to tests builds and so on) The only issue with this approach can be services / software that has hard dependencies on specific cloud products you can’t run “locally” but even in that case you can always spin up / tear down super small versions of them to use for the setup. Python has been great to glue orchestrating everything with easy to use kubernetes and docker libraries, and aws/azure cli commands as needed. If you have a product that you can’t easily build or deploy in an automated manner you’re going to have a bad time at disaster recovery time or when the one person that knew how that one piece gets installed is laid off.
fuzzy2 2y ago
Wow that sounds like the absolute worst. My condolences. In projects I manage I even try not to use containers for the service being developed. (Only for its dependencies.) Everything that affects cycle times must go. That being said, you should also have a dev environment. QA isn't for development. It'll certainly be cheaper than firing people.
- Pxtl 2y ago
  Right? I do the DevOps/architect stuff at my org and one game changer for testing was configuring it to spin up a separate instance per pull-request. Occasionally things get wonky because all the PR instances share an authentication/authorization server, but by and large it is excellent for being able to quickly demo. But it does mean that you have to be religious about building links and connection-strings in your code.
  - rollcat 2y ago
    Once you have 3 environments (which IMHO is the minimum viable number: dev, staging, prod), it tends to be much easier to generalise that to N, at least from the application side (spinning up the extra infra is a separate beast). It tends to weed out many of the "if (prod)"s because there should now be differences between dev & staging that you need to take care of, and sprinkling elaborate switch statements gets tedious really fast - most devs tend to be happier actually simplifying code. I've seen some really nasty offenders though, e.g. actual business logic hidden behind an "if (window.location.hostname == ...)". Sometimes it takes a lot of patience to explain that this is how you get production-only bugs that QA will never have a chance to find or reproduce.
    - Pxtl 2y ago
      For us the challenge was our DNS wasn't set up to spin up unlimited subdomains, so we had to run applications on dev.companyname.tld/application_prInstanceID/, so that was the case where we had to make application-level changes to support N instances, since some code assumed they would have their own private sub-domain-name.
patrakov 2y ago
I have also seen a prohibition to have a local copy of the code coming from cybersecurity guys worried about laptops being compromised, for example, through an email with a malicious link. "Here is a web-based editor; please develop everything there, and at the bottom, there is a terminal to run your application," they say.
- acdha 2y ago
  Yes - that’s an indictment of the security group on multiple levels (Kerckhoffs's principle, failure to deploy FDE / MFA, etc.) but depending on the organizational culture the cost might be shifted to the development groups and not everyone has senior technical staff able to challenge it.

rcxdude 2y ago

Yeah, this is the situation being argued against. If you don't focus on making your environment reproducible, you wind up tangled in a mess no-one can understand. I imagine you also don't expect to be able to get things running again after a disaster strikes.

tetraodonpuffer 2y ago

I don’t understand some of my colleagues that feel this way either, I’ve been around a long time but having a way to set up a ”local” environment with a local automated build to me seems priority one for any piece of software cloud or otherwise. By “local” I mean self contained, my “local” environment is right now a single cloud Linux VM with kind for kubernetes, all sorts of k8s local monitoring / cert issuing / stubbed things, metallb and openvpn to access (with a local to my computer dhcpcd for DNS resolution) and so on, all building from local and orchestrated by a python set of scripts. All of this works great and has saved me so many times and made development so much faster compared to teams that rely on ci/cd even for basic development (where every commit takes an hour+ to deploy due to tests builds and so on) The only issue with this approach can be services / software that has hard dependencies on specific cloud products you can’t run “locally” but even in that case you can always spin up / tear down super small versions of them to use for the setup. Python has been great to glue orchestrating everything with easy to use kubernetes and docker libraries, and aws/azure cli commands as needed. If you have a product that you can’t easily build or deploy in an automated manner you’re going to have a bad time at disaster recovery time or when the one person that knew how that one piece gets installed is laid off.

fuzzy2 2y ago

Wow that sounds like the absolute worst. My condolences. In projects I manage I even try not to use containers for the service being developed. (Only for its dependencies.) Everything that affects cycle times must go. That being said, you should also have a dev environment. QA isn't for development. It'll certainly be cheaper than firing people.

Pxtl 2y ago
Right? I do the DevOps/architect stuff at my org and one game changer for testing was configuring it to spin up a separate instance per pull-request. Occasionally things get wonky because all the PR instances share an authentication/authorization server, but by and large it is excellent for being able to quickly demo. But it does mean that you have to be religious about building links and connection-strings in your code.
- rollcat 2y ago
  Once you have 3 environments (which IMHO is the minimum viable number: dev, staging, prod), it tends to be much easier to generalise that to N, at least from the application side (spinning up the extra infra is a separate beast). It tends to weed out many of the "if (prod)"s because there should now be differences between dev & staging that you need to take care of, and sprinkling elaborate switch statements gets tedious really fast - most devs tend to be happier actually simplifying code. I've seen some really nasty offenders though, e.g. actual business logic hidden behind an "if (window.location.hostname == ...)". Sometimes it takes a lot of patience to explain that this is how you get production-only bugs that QA will never have a chance to find or reproduce.
  - Pxtl 2y ago
    For us the challenge was our DNS wasn't set up to spin up unlimited subdomains, so we had to run applications on dev.companyname.tld/application_prInstanceID/, so that was the case where we had to make application-level changes to support N instances, since some code assumed they would have their own private sub-domain-name.

patrakov 2y ago

I have also seen a prohibition to have a local copy of the code coming from cybersecurity guys worried about laptops being compromised, for example, through an email with a malicious link. "Here is a web-based editor; please develop everything there, and at the bottom, there is a terminal to run your application," they say.

acdha 2y ago
Yes - that’s an indictment of the security group on multiple levels (Kerckhoffs's principle, failure to deploy FDE / MFA, etc.) but depending on the organizational culture the cost might be shifted to the development groups and not everyone has senior technical staff able to challenge it.