4 ms·

Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big th

Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. And having a different development machine than production leads to completely predictable and unavoidable problems. Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing.

hosh 2y ago
I have used remote dev machines just fine, but my workflow vastly differs from many of my coworkers: terminal-only spacemacs + tmux + mosh. I have a lot of CLI and TUI tools, and I do not use VScode at all. The main GUI app I run is a browser, and that runs locally. I have worked on developing VMs for other developers that rely on a local IDE such. The main sticking point is syncing and schlepping source code (something my setup avoids because the source code and editor is on the remote machine). I have tried a number of approaches, and I sympathize with the article author. So, in response to "Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing." <-- syncing and schlepping source code is by no means a solved problem. I can also say that, my spacemacs config is very vanilla. Like my phone, I don't want to be messing with it when I want to code. Writing tooling for my editor environment is a sideshow for the work I am trying to finish.
- pjmlp 2y ago
  Me as well, specially in the days that there was only a UNIX dev server for everyone. It was never an issue to use X Windows on them, with hummingbird on my Windows thin client. I guess a new generation has to learn the ways of timesharing development.
- xcrunner529 2y ago
  I am hardly a dev but occasionally have had to do some or some scripting or web stuff and have really loved VSCode and using the remote SSH support to basically feel like I’m coding locally. Does that not work for your devs?
  - djbusby 2y ago
    That puts a "scary" VSCode blob on the remote-server. Some orgs do not like that, even if it's a "work" class box.
  - pjmlp 2y ago
    UNIX, and other competing timesharing systems of the time, have always been remote first, with Windows catching up with Citrix, followed by RDP, and nowadays finally headless as well. Nowadays Web frontends and SSH/cloud shell, have replaced what used to be X Windows / telnet / rsh, but the underlying workflows aren't much different than running an IDE / emacs /vi / joe /... from a UNIX development server in a 1990's office.
    - 0xbadcafebee 2y ago
      The funniest (?) thing to me about all this: we're still hoping, if we do things right, to replicate the technology (terminals) from 50 years ago. I honestly don't understand why nobody has simply invented some software to solve this problem, after 50 years.
      - pjmlp 2y ago
        Add to it the whole TUI fashion, as if we weren't doing them already 50 years ago. We did, it is called GUI and language REPLs, like Smalltalk and Interlisp-D development enviroments, with graphical based terminals, not dependent on replicating virtual teletypes. Still something that seems problematic to take off the way it should.
kgeist 2y ago
We have a project which spawns around 80 Docker containers and runs pretty OK on a 5 year old Dell laptop with 16GB RAM. The fans run crazy and the laptop is always very hot but I haven't noticed considerable lags, even with IntelliJ running. Most services are written in Go though and are pretty lightweight.
- Tainnor 2y ago
  > Most services are written in Go though and are pretty lightweight That's probably the difference. Throw elasticsearch, kafka and a bunch of Java services in and you'll be easily exhausting your RAM (at least at startup).
  - stephenr 2y ago
    16GB of RAM is hardly a gargantuan amount in 2024 - it's not unreasonable to expect someone running a local dev environment to have a more practical amount. I wouldn't buy (or recommend) any machine for dev work with less than 64GB in 2024.
    - kgeist 2y ago
      >I wouldn't buy (or recommend) any machine for dev work with less than 64GB in 2024. Honestly, I think it depends on what kind of languages/frameworks/tools you use. Go is very lightweight, and you can safely run hundreds of Go services under 16 GB no problem (and with a crappy CPU, too). Python/Java/PHP etc. on the other hand, are much more wasteful. Our Go shop only now is considering maybe buying 32GB dev machines...
  - kgeist 2y ago
    It has OpenSearch, RabbitMQ, Redis, MySQL
arp242 2y ago
> the stack always grows to the point that a dev can no longer test it on their own machine So the solution here is to not have that kind of "stack". I mean, if it's all so big and complex that it can't be run on a laptop then you almost certainly got a lot of problems regardless. What typically happens is tons of interconnected services without clear abstractions or interfaces, and no one really understands this spaghetti mess, and people just keep piling crap on top of it. This leads to all sorts of problems. Everywhere I've seen this happen they had real problems running stuff in production too, because it was a complex spaghetti mess. The abstracted "easy" dev-env (in whatever form that came) is then also incredibly complex, finicky, and brittle. Never mind running tests, which is typically even worse. It's not uncommon for it all to be broken for every other new person who joins because changes somewhere broke the setup steps which are only run for new people. Everyone else is afraid to do anything with their machine "because it now works'. There are some exceptions where you really need a big beefy machine for a dev env and tests, maybe, but they're few and far between.
- fragmede 2y ago
  > So the solution here is to not have that kind of "stack". Reminds me of my favorites debugging technique. It's super fast: Don't write any bugs!
  - vinnymac 2y ago
    Code is a liability. Not writing code is one of the best things a developer can do on any particular day, aside of course from deleting it ;)
  - halfcat 2y ago
    Quite an effective approach: https://github.com/kelseyhightower/nocode https://github.com/kelseyhightower/nocode
  - arp242 2y ago
    What a boring trite reply. All of this is analogous to badly written spaghetti code. And yes, you can absolutely avoid all of this if you know what you're doing.
maxrecursion 2y ago
"Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine" Isn't this problem solved by CICD? When the developer is ready to test, they make a commit, and the pipeline deploys the code to a dev/test environment. That's how my teams have been doing it.
- jrockway 2y ago
  This turns a 1 hour task into a 1 day task. Fast feedback cycles are critical to software development. I don't quite understand how people get into the situation where their work can't fit on their workstation. I've worked on huge projects at huge tech companies, and I could run everything on my workstation. I've worked at startups where the CI situation was passing 5% of the time and required 3 hours to run, that you can now run on your workstation in seconds. What you do is fix the stuff that doesn't fit. The most insidious source of slowness I've encountered is tests that use test databases set to fsync = on. This severely limits parallelism and speed in a way that's difficult to diagnose; you have plenty of CPU and memory available, but the tests just aren't going very fast. (I don't remember how I stumbled upon this insight. I think I must have straced Postgres and been like "ohhhhhhhhh, of course".)
  - d_sem 2y ago
    It's likely you haven't come across these use cases in your professional career, but I assure you its very common. My entire career has only seen projects where you need dozen to hundreds of CPU's in order to have a short feedback loop to verify the system works. I saw this in simple algorithms in automotive, to Advanced Driver Assistance Systems and machine learning applications. When you are working on a software project that has 1,000 active developers checking in code daily and require a stable system build you need lots of compute.
    - baq 2y ago
      There's a lot of folks in startups who think 100 devs is a large org and can't comprehend the scale at which '100% tests pass' stops being a build blocker. I've migrated from such an org to a late stage startup and 'tests must pass' even if fifty engineers are blocked with their PRs and the release train is fully halted. 'But our pipelines must be green' no they don't, at least not all of them.
pylua 2y ago
Really? I can't imagine not running the code locally. Honestly, my company has a micro services architecture, and I will just comment out the docker-compose pieces that I am not using. If I am developing/testing a particular component then I will enable it. How tightly coupled are these systems?
rossjudson 2y ago
The stack(factory) must grow.
lmm 2y ago
> Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. It doesn't have to be like that. I've worked on a 10MLOC codebase with 500+ committers - all perfectly runnable locally, on admittedly slightly beefy dev machines. It's true that systems will grow without limit unless some force exists to counter this, but keeping your stack something you can sanely run on a development machine is well worth spending some actual effort on.
- 0xbadcafebee 2y ago
  It depends on the application. I've worked at places where the main application being written needs 128GB to run. Another where it's computationally intense and will take hours for a single run on anything but the most massive machines. Another where the dataset is huge and simply being remote makes the bandwidth requirements untenable unless you live across from a data center. I've seen teams just accept their workflow will take 15 minutes each time they run their app because they're copying stuff from remote each time. Remote dev gives you as beefy a machine as you want, with the fastest network and lowest bandwidth. But that's only the resource problem. Another problem I have seen my entire career, is devs can't keep their machines configured the same. They have different model laptops, they don't pin their app versions, they configure and install things by hand. Each time they change something by accident it takes them hours, days, sometimes weeks, to get it working again. That also can lead to bugs developing the app, which wastes a huge amount of time. And then there's the fact that their local copy runs completely differently than it does in production. This leads to the app being written with certain assumptions about how it runs, that turn out to be false in production. I've seen this lead to catastrophe, as well as just weeks to months of wasted time, trying to track down issues. This is an undeniable, existential issue. Finally, it's rare for local setups to be secure. Often devs get too much access from their local machines, and this is stolen by infostealer malware and compromise happens. A protected remote environment is easier to secure. A lot of development is hampered by all the crappy corporate security tools that's on laptops now. Remote dev allows you to bypass all that and have a fully working yet protected network without restrictions. Is remote dev a pain? Right now, yeah, because nobody has made it be less painful. So of course it's easier on the local machine. But it's not ideal. I'm sure eating with a spoon was more painful than eating with your hands, until forks were popularized in the 18th century. Change took a long time, but I think most of us prefer the change once new tools became widely available.
stephenr 2y ago
> the stack always grows to the point that a dev can no longer test it on their own machine Sounds like you have a different problem. CPU resources required to run your stack should be very minimal if it's a single user accessing it for local testing idle threads don't consume oodles of cpu cycles to do nothing. Memory use may be significant even in that case (depending on your stack) but let's be realistic. If your stack is so large that it alone requires more memory than a dev machine can spare with an IDE open, the cost of providing developers with capable workstations will pale in comparison to the cost of running the prod environment. I have a client whose prod environment is 2x load balancer; 2x app server; 3x DB cluster node - all rented virtual machines. We just upgraded to higher spec machines to give headroom over the next couple of years (ie most machines doubled the RAM from the previous generation). My old workstation bought in 2018 had enough memory that it could virtualise the current prod environment with the same amounts of RAM as prod, and still have 20GB free. My current workstation would have 80+ GB free. In 95% of cases if you can't run the stack for a single user testing it, on a single physical machine, you're doing something drastically wrong somewhere.

hosh 2y ago

I have used remote dev machines just fine, but my workflow vastly differs from many of my coworkers: terminal-only spacemacs + tmux + mosh. I have a lot of CLI and TUI tools, and I do not use VScode at all. The main GUI app I run is a browser, and that runs locally. I have worked on developing VMs for other developers that rely on a local IDE such. The main sticking point is syncing and schlepping source code (something my setup avoids because the source code and editor is on the remote machine). I have tried a number of approaches, and I sympathize with the article author. So, in response to "Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing." <-- syncing and schlepping source code is by no means a solved problem. I can also say that, my spacemacs config is very vanilla. Like my phone, I don't want to be messing with it when I want to code. Writing tooling for my editor environment is a sideshow for the work I am trying to finish.

pjmlp 2y ago
Me as well, specially in the days that there was only a UNIX dev server for everyone. It was never an issue to use X Windows on them, with hummingbird on my Windows thin client. I guess a new generation has to learn the ways of timesharing development.
xcrunner529 2y ago
I am hardly a dev but occasionally have had to do some or some scripting or web stuff and have really loved VSCode and using the remote SSH support to basically feel like I’m coding locally. Does that not work for your devs?
- djbusby 2y ago
  That puts a "scary" VSCode blob on the remote-server. Some orgs do not like that, even if it's a "work" class box.
- pjmlp 2y ago
  UNIX, and other competing timesharing systems of the time, have always been remote first, with Windows catching up with Citrix, followed by RDP, and nowadays finally headless as well. Nowadays Web frontends and SSH/cloud shell, have replaced what used to be X Windows / telnet / rsh, but the underlying workflows aren't much different than running an IDE / emacs /vi / joe /... from a UNIX development server in a 1990's office.
  - 0xbadcafebee 2y ago
    The funniest (?) thing to me about all this: we're still hoping, if we do things right, to replicate the technology (terminals) from 50 years ago. I honestly don't understand why nobody has simply invented some software to solve this problem, after 50 years.
    - pjmlp 2y ago
      Add to it the whole TUI fashion, as if we weren't doing them already 50 years ago. We did, it is called GUI and language REPLs, like Smalltalk and Interlisp-D development enviroments, with graphical based terminals, not dependent on replicating virtual teletypes. Still something that seems problematic to take off the way it should.

kgeist 2y ago

We have a project which spawns around 80 Docker containers and runs pretty OK on a 5 year old Dell laptop with 16GB RAM. The fans run crazy and the laptop is always very hot but I haven't noticed considerable lags, even with IntelliJ running. Most services are written in Go though and are pretty lightweight.

Tainnor 2y ago
> Most services are written in Go though and are pretty lightweight That's probably the difference. Throw elasticsearch, kafka and a bunch of Java services in and you'll be easily exhausting your RAM (at least at startup).
- stephenr 2y ago
  16GB of RAM is hardly a gargantuan amount in 2024 - it's not unreasonable to expect someone running a local dev environment to have a more practical amount. I wouldn't buy (or recommend) any machine for dev work with less than 64GB in 2024.
  - kgeist 2y ago
    >I wouldn't buy (or recommend) any machine for dev work with less than 64GB in 2024. Honestly, I think it depends on what kind of languages/frameworks/tools you use. Go is very lightweight, and you can safely run hundreds of Go services under 16 GB no problem (and with a crappy CPU, too). Python/Java/PHP etc. on the other hand, are much more wasteful. Our Go shop only now is considering maybe buying 32GB dev machines...
- kgeist 2y ago
  It has OpenSearch, RabbitMQ, Redis, MySQL

arp242 2y ago

> the stack always grows to the point that a dev can no longer test it on their own machine So the solution here is to not have that kind of "stack". I mean, if it's all so big and complex that it can't be run on a laptop then you almost certainly got a lot of problems regardless. What typically happens is tons of interconnected services without clear abstractions or interfaces, and no one really understands this spaghetti mess, and people just keep piling crap on top of it. This leads to all sorts of problems. Everywhere I've seen this happen they had real problems running stuff in production too, because it was a complex spaghetti mess. The abstracted "easy" dev-env (in whatever form that came) is then also incredibly complex, finicky, and brittle. Never mind running tests, which is typically even worse. It's not uncommon for it all to be broken for every other new person who joins because changes somewhere broke the setup steps which are only run for new people. Everyone else is afraid to do anything with their machine "because it now works'. There are some exceptions where you really need a big beefy machine for a dev env and tests, maybe, but they're few and far between.

fragmede 2y ago
> So the solution here is to not have that kind of "stack". Reminds me of my favorites debugging technique. It's super fast: Don't write any bugs!
- vinnymac 2y ago
  Code is a liability. Not writing code is one of the best things a developer can do on any particular day, aside of course from deleting it ;)
- halfcat 2y ago
  Quite an effective approach: https://github.com/kelseyhightower/nocode https://github.com/kelseyhightower/nocode
- arp242 2y ago
  What a boring trite reply. All of this is analogous to badly written spaghetti code. And yes, you can absolutely avoid all of this if you know what you're doing.

maxrecursion 2y ago

"Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine" Isn't this problem solved by CICD? When the developer is ready to test, they make a commit, and the pipeline deploys the code to a dev/test environment. That's how my teams have been doing it.

jrockway 2y ago
This turns a 1 hour task into a 1 day task. Fast feedback cycles are critical to software development. I don't quite understand how people get into the situation where their work can't fit on their workstation. I've worked on huge projects at huge tech companies, and I could run everything on my workstation. I've worked at startups where the CI situation was passing 5% of the time and required 3 hours to run, that you can now run on your workstation in seconds. What you do is fix the stuff that doesn't fit. The most insidious source of slowness I've encountered is tests that use test databases set to fsync = on. This severely limits parallelism and speed in a way that's difficult to diagnose; you have plenty of CPU and memory available, but the tests just aren't going very fast. (I don't remember how I stumbled upon this insight. I think I must have straced Postgres and been like "ohhhhhhhhh, of course".)
- d_sem 2y ago
  It's likely you haven't come across these use cases in your professional career, but I assure you its very common. My entire career has only seen projects where you need dozen to hundreds of CPU's in order to have a short feedback loop to verify the system works. I saw this in simple algorithms in automotive, to Advanced Driver Assistance Systems and machine learning applications. When you are working on a software project that has 1,000 active developers checking in code daily and require a stable system build you need lots of compute.
  - baq 2y ago
    There's a lot of folks in startups who think 100 devs is a large org and can't comprehend the scale at which '100% tests pass' stops being a build blocker. I've migrated from such an org to a late stage startup and 'tests must pass' even if fifty engineers are blocked with their PRs and the release train is fully halted. 'But our pipelines must be green' no they don't, at least not all of them.

pylua 2y ago

Really? I can't imagine not running the code locally. Honestly, my company has a micro services architecture, and I will just comment out the docker-compose pieces that I am not using. If I am developing/testing a particular component then I will enable it. How tightly coupled are these systems?

rossjudson 2y ago

The stack(factory) must grow.

lmm 2y ago

> Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. It doesn't have to be like that. I've worked on a 10MLOC codebase with 500+ committers - all perfectly runnable locally, on admittedly slightly beefy dev machines. It's true that systems will grow without limit unless some force exists to counter this, but keeping your stack something you can sanely run on a development machine is well worth spending some actual effort on.

0xbadcafebee 2y ago
It depends on the application. I've worked at places where the main application being written needs 128GB to run. Another where it's computationally intense and will take hours for a single run on anything but the most massive machines. Another where the dataset is huge and simply being remote makes the bandwidth requirements untenable unless you live across from a data center. I've seen teams just accept their workflow will take 15 minutes each time they run their app because they're copying stuff from remote each time. Remote dev gives you as beefy a machine as you want, with the fastest network and lowest bandwidth. But that's only the resource problem. Another problem I have seen my entire career, is devs can't keep their machines configured the same. They have different model laptops, they don't pin their app versions, they configure and install things by hand. Each time they change something by accident it takes them hours, days, sometimes weeks, to get it working again. That also can lead to bugs developing the app, which wastes a huge amount of time. And then there's the fact that their local copy runs completely differently than it does in production. This leads to the app being written with certain assumptions about how it runs, that turn out to be false in production. I've seen this lead to catastrophe, as well as just weeks to months of wasted time, trying to track down issues. This is an undeniable, existential issue. Finally, it's rare for local setups to be secure. Often devs get too much access from their local machines, and this is stolen by infostealer malware and compromise happens. A protected remote environment is easier to secure. A lot of development is hampered by all the crappy corporate security tools that's on laptops now. Remote dev allows you to bypass all that and have a fully working yet protected network without restrictions. Is remote dev a pain? Right now, yeah, because nobody has made it be less painful. So of course it's easier on the local machine. But it's not ideal. I'm sure eating with a spoon was more painful than eating with your hands, until forks were popularized in the 18th century. Change took a long time, but I think most of us prefer the change once new tools became widely available.

stephenr 2y ago

> the stack always grows to the point that a dev can no longer test it on their own machine Sounds like you have a different problem. CPU resources required to run your stack should be very minimal if it's a single user accessing it for local testing idle threads don't consume oodles of cpu cycles to do nothing. Memory use may be significant even in that case (depending on your stack) but let's be realistic. If your stack is so large that it alone requires more memory than a dev machine can spare with an IDE open, the cost of providing developers with capable workstations will pale in comparison to the cost of running the prod environment. I have a client whose prod environment is 2x load balancer; 2x app server; 3x DB cluster node - all rented virtual machines. We just upgraded to higher spec machines to give headroom over the next couple of years (ie most machines doubled the RAM from the previous generation). My old workstation bought in 2018 had enough memory that it could virtualise the current prod environment with the same amounts of RAM as prod, and still have 20GB free. My current workstation would have 80+ GB free. In 95% of cases if you can't run the stack for a single user testing it, on a single physical machine, you're doing something drastically wrong somewhere.