33 ms·

This sounds great. Certainly the IP zealots love it because the code never leaves the walled garden, and it’s a super-secure blah blah environment. Consistent t

by teeray 4y ago

This sounds great. Certainly the IP zealots love it because the code never leaves the walled garden, and it’s a super-secure blah blah environment. Consistent tooling, easy onboarding… you get the idea. All good until you have an outage. Then your development team’s productivity drops to exactly zero while you fix it and your entire production environment is now potentially vulnerable to defects you can’t fix until the development environment is fixed (better hope it stays running). This is when companies realize that the development environment is actually a service and it needs higher SLA targets than production, but it will never get the attention that it needs to achieve those targets (because it’s just dev, right?).

gandalfgeek 4y ago
Every service has the risk of introducing a new point of failure, of course. But compare dev productivity lost due to service downtime with that of each new SWE in your org burning time to A) setup their own unique snowflake of an environment and B) futzing and debugging it when it breaks or there's a software update.
- erik_seaberg 4y ago
  > burning time to setup their own unique snowflake of an environment Anyone good is doing that even if you tell him not to. One size does not fit all.
charcircuit 4y ago
You already have the same concern with source control, code review, continuous integration, continuous deployment, etc. Dev servers are arguably much simpler to provide without issues.
- teeray 4y ago
  But I don’t really have those concerns—they’re better mitigated. If source control goes down, I have a complete clone on my machine and so do my coworkers (yay git). I have done code reviews by pasting `git am` formatted patches between coworkers during particularly long GitHub outages. CI/CD is more problematic, but I only need those things to integrate work. I can still do work and verify correctness with tests that I can run locally. None of that is possible in a centrally-managed dev VM setup. When the VMs go down, you send your dev team home until it’s fixed. You’re still paying their salaries while you pay another team to make them productive again.
  - fragmede 4y ago
    Depends on the scale you're working with. If your complete code can fit locally, you're working at a different scale. Huge monorepos won't fit on a laptop-sized local disk these days, and a full local dev env would take a half dozen VMs to even replicate half-successfully.
    - teeray 4y ago
      We’re talking generally here about all software development, so of course we can find exceptional cases to refute anything anyone says. Let’s scope the discussion to what is likely a common developer scenario where `git clone` works and it’s possible to run a reasonable facsimile of the application locally (maybe with some simulators for cloud services).
      - jonas21 4y ago
        The article we're discussing is from Uber, which has a huge monorepo. It's exactly the use case they designed Devpod for.
    - raverbashing 4y ago
      > Huge monorepos won't fit on a laptop-sized local disk these days Reason 743 why monorepos create more problems than they solve
  - charcircuit 4y ago
    Nothing is stopping developers from running the docker image locally.
cyc116 4y ago
Not sure what's done at Uber but I imagine HA is a requirement. There are mitigations and a couple of hours of dev downtime is imo not the end of world. Sure, prod incident could overlap with a devpod incident but sounds like they still have the choice to do local dev.
gtirloni 4y ago
*> higher SLA targets than production Lost revenue is usually more impactful than the occasional lost developer time.
unity1001 4y ago
> All good until you have an outage Better to have hundreds, or * gasp * thousands of engineers having to fix local package dependency problems on their computers every other day. Much better than those engineers having to keep that gigantic context loaded in their mind so that they fix those local problems. That effort spent for maintaining local environments and navigating the package management hell is effort not being spent on creating actual code.
- eloisius 4y ago
  That time is spread randomly over thousands of devs and not lost all at once during a sev1
  - unity1001 4y ago
    That time is spread randomly over thousands of devs and each of them encounter similar package dependency issues every other day. Its not 1 in 1000 devs every day. Everyone will need to solve it locally themselves.

by teeray 4y ago

gandalfgeek 4y ago

Every service has the risk of introducing a new point of failure, of course. But compare dev productivity lost due to service downtime with that of each new SWE in your org burning time to A) setup their own unique snowflake of an environment and B) futzing and debugging it when it breaks or there's a software update.

erik_seaberg 4y ago
> burning time to setup their own unique snowflake of an environment Anyone good is doing that even if you tell him not to. One size does not fit all.

charcircuit 4y ago

You already have the same concern with source control, code review, continuous integration, continuous deployment, etc. Dev servers are arguably much simpler to provide without issues.

teeray 4y ago
But I don’t really have those concerns—they’re better mitigated. If source control goes down, I have a complete clone on my machine and so do my coworkers (yay git). I have done code reviews by pasting `git am` formatted patches between coworkers during particularly long GitHub outages. CI/CD is more problematic, but I only need those things to integrate work. I can still do work and verify correctness with tests that I can run locally. None of that is possible in a centrally-managed dev VM setup. When the VMs go down, you send your dev team home until it’s fixed. You’re still paying their salaries while you pay another team to make them productive again.
- fragmede 4y ago
  Depends on the scale you're working with. If your complete code can fit locally, you're working at a different scale. Huge monorepos won't fit on a laptop-sized local disk these days, and a full local dev env would take a half dozen VMs to even replicate half-successfully.
  - teeray 4y ago
    We’re talking generally here about all software development, so of course we can find exceptional cases to refute anything anyone says. Let’s scope the discussion to what is likely a common developer scenario where `git clone` works and it’s possible to run a reasonable facsimile of the application locally (maybe with some simulators for cloud services).
    - jonas21 4y ago
      The article we're discussing is from Uber, which has a huge monorepo. It's exactly the use case they designed Devpod for.
  - raverbashing 4y ago
    > Huge monorepos won't fit on a laptop-sized local disk these days Reason 743 why monorepos create more problems than they solve
- charcircuit 4y ago
  Nothing is stopping developers from running the docker image locally.

cyc116 4y ago

Not sure what's done at Uber but I imagine HA is a requirement. There are mitigations and a couple of hours of dev downtime is imo not the end of world. Sure, prod incident could overlap with a devpod incident but sounds like they still have the choice to do local dev.

gtirloni 4y ago

*> higher SLA targets than production Lost revenue is usually more impactful than the occasional lost developer time.

unity1001 4y ago

> All good until you have an outage Better to have hundreds, or * gasp * thousands of engineers having to fix local package dependency problems on their computers every other day. Much better than those engineers having to keep that gigantic context loaded in their mind so that they fix those local problems. That effort spent for maintaining local environments and navigating the package management hell is effort not being spent on creating actual code.

eloisius 4y ago
That time is spread randomly over thousands of devs and not lost all at once during a sev1
- unity1001 4y ago
  That time is spread randomly over thousands of devs and each of them encounter similar package dependency issues every other day. Its not 1 in 1000 devs every day. Everyone will need to solve it locally themselves.