15 ms·

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, an

Personally - just let the developer own the machine they use for development. If you really need consistency for the environment - Let them own the machine, and then give them a stable base VM image, and pay for decent virtualization tooling that they run... on their own machine. I have seen several attempts to move dev environments to a remote host. They invariably suck. Yes - that means you need to pay for decent hardware for your devs, it's usually cheaper than remote resources (for a lot of reasons). Yes - that means you need to support running your stack locally. This is a good constraint (and a place where containers are your friend for consistency). Yes - that means you need data generation tooling to populate a local env. This can be automated relatively well, and it's something you need with a remote env anyways. --- The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). I'm my experience, the vast majority of companies should worry less about this - your value as a company isn't your source code in 99.5% of cases, it's the team that executes that source code in production. If you're in the 0.5% of other cases... you know it and you should be in an air-gapped closed room anyways (and I've worked in those too...)

dangsux 2y ago
[dead]
shriek 2y ago
And the reason they suck is the feedback loop is just too high as compared to running it locally. You have to jump through hoops to debug/troubleshoot your code or any issues that you come across between your code and output of your code. And it's almost impossible to work on things when you have spotty internet. I haven't worked on extremely sensitive data but for PII data from prod to dev, scrubbing is a good practice to follow. This will vary based on the project/team you're on of course.
- ethbr1 2y ago
  Aka 'if a developer knew beforehand everything they needed, it wouldn't be development'
- marcosdumay 2y ago
  That's the least important problem. The developers also lack knowledge about the environment; can't evolve the environment; can't test the environment for bugs; and invariably interfere with each other because it's never isolated well. And also, yes, it adds lag. Anyway, yes, working locally on false data that little resemblance to production still beats remote environments.
binary132 2y ago
Hello. Currently debugging my kubernetes-based dev pod and not getting anything else done. What fun!
deleted 2y ago
[deleted]
haolez 2y ago
Sometimes I don't even use virtual envs when developing locally in Python. I just install everything that I need with pip --user and be done with it. Never had any conflicts with system packages whatsoever. If I somehow break my --user environment, I simply delete it and start again. Never had any major version mismatch in dependencies between my machine and what was running in production. At least not anything that would impact the actual task that I was working on. I'm not recommending this as a best practice. I just believe that we, as developers, end up creating some myths to ourselves of what works and what doesn't. It's good to re-evaluate these beliefs now and then.
- __MatrixMan__ 2y ago
  When doing this re-evaluation, please consider that others might be quietly working very hard to discover and recreate locally whatever secret sauce you and production share.
- ctippett 2y ago
  I'm not going to second-guess what works for you, but Python makes it so easy to work with an ephemeral environment. python -m venv .venv
  - haolez 2y ago
    Yeah, I know. But then you have to make sure that your IDE is using the correct environment, that the notebook is using the correct environment, that the debugger is using the correct environment. It's trivial to setup a venv, but sometimes it's just not worth it for me.
    - zo1 2y ago
      This is one of the main reasons I tell people not to use VSCode. The people most likely to use it are juniors and people new to python specifically, and they're the most likely to fall victim to 'but my "IDE" says it's running 3.8 with everything installed, but when I run it from my terminal it's a different python 3.8' I watched it last week. With 4 (I hope junior) Devs in a "pair programming" session that forced me to figure out how VSCode does virtual envs, and still I had to tell them like 3 times "stop opening a damn new terminal, it's obviously not setup with our python version, run the command inside the one that has the virtual env activated".
csweichel 2y ago
OP here. There definitely is a place for running things on your local machine. Exactly as you say: one can get a great deal of consistency using VMs. One of the benefits of moving away from Kubernetes, to a runner-based architecture , is that we can now seamlessly support cloud-based and local environments (https://www.gitpod.io/blog/introducing-gitpod-desktop https://www.gitpod.io/blog/introducing-gitpod-desktop). What's really nice about this is that with this kind of integration there's very little difference in setting up a dev env in the cloud or locally. The behaviour and qualities of those environments can differ vastly though (network bandwidth, latency, GPU, RAM, CPUs, ARM/x86).
- edplmajor 2y ago
  Need middleware libs that react to eBPF data and signal app code to scale up/down forks in their own memory VM, like V8 Kubernetes is another mess of userspace ops tools. Userspace is for composable UI not backend. Kube and Chef and all those other ops tools are backend functionality being used like UI by leet haxxors
- master_crab 2y ago
  Hi Christian. We just deployed Gitpod EKS at our company in NY. Can we get some details on the replacement architecture? I’m sure it’s great but the devil is always in the details.
- michaelt 2y ago
  > The behaviour and qualities of those environments can differ vastly though (network bandwidth, latency, GPU, RAM, CPUs, ARM/x86). For example, when you're running on your local machine you've actually got the amount of RAM and CPU advertised :)
  - sethammons 2y ago
    "Hm, why does my Go service on a pod with 2.2 cpu's think it has 6k? Oh, it thinks it has the whole cluster. Nice; that is why scheduling has been an issue"
- justinclift 2y ago
lotharcable 2y ago
I strongly recommend just switching the Dev environment over to Linux and taking advantage of tools like "distrobox" and "toolbx". https://github.com/89luca89/distrobox https://github.com/89luca89/distrobox https://containertoolbx.org/ https://containertoolbx.org/ It is sorta like Vagrant, but instead of using virtualbox virtual machines you use podman containers. This way you get to use OCI images for your "dev environment" that integrates directly into your desktop. https://podman.io/ https://podman.io/ There is some challenges related to usermode networking for non-root-managed controllers and desktop integration has some additional complications. But besides that it has almost no overhead and you can have unfettered access to things like GPUs. Also it is usually pretty easy to convert your normal docker or kubernetes containers over to something you can run on your desktop. Also it is possible to use things like Kubernetes pods definitions to deploy sets of containers with podman and manage it with systemd and such things. So you can have "clouds of containers" that your dev container needs access to locally. If there is a corporate need for window-specific applications then running Windows VMs or doing remote applications over RDP is a possible work around. If everything you are targeting as a deployment is going to be Linux-everything then it doesn't make a lot of sense to jump through a bunch of hoops and cause a bunch of headaches just to avoid having it as workstation OS.
- trog 2y ago
  If you're doing this, there are many cases where you might as well just spin up a decent Linux server and give your developers accounts on that? With some pretty basic setup everyone can just run their own stuff within their own user account. You'll run into occasional issues (e.g. if everyone is trying to run default node.js on default port) but with some basic guardrails it feels like it should be OK? I'm remembering back to when my old company ran a lot of PHP projects. Each user just had their own development environment and their own Apache vhost. They wrote their code and tested it in their own vhost. Then we'd merge to a single separate vhost for further testing. I am trying to remember anything about what was painful about it but it all basically Just Worked. Everyone had remote access via VPN; the worst case scenario for them was they'd have to work from home with a bit of extra latency.
  - 2y ago
idunnoman1222 2y ago
Sounds like you are not using a lot of hardware - Rfid, POS, top-spec video cards, etc
0xbadcafebee 2y ago
Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. And having a different development machine than production leads to completely predictable and unavoidable problems. Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing.
- hosh 2y ago
  I have used remote dev machines just fine, but my workflow vastly differs from many of my coworkers: terminal-only spacemacs + tmux + mosh. I have a lot of CLI and TUI tools, and I do not use VScode at all. The main GUI app I run is a browser, and that runs locally. I have worked on developing VMs for other developers that rely on a local IDE such. The main sticking point is syncing and schlepping source code (something my setup avoids because the source code and editor is on the remote machine). I have tried a number of approaches, and I sympathize with the article author. So, in response to "Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing." <-- syncing and schlepping source code is by no means a solved problem. I can also say that, my spacemacs config is very vanilla. Like my phone, I don't want to be messing with it when I want to code. Writing tooling for my editor environment is a sideshow for the work I am trying to finish.
  - pjmlp 2y ago
    Me as well, specially in the days that there was only a UNIX dev server for everyone. It was never an issue to use X Windows on them, with hummingbird on my Windows thin client. I guess a new generation has to learn the ways of timesharing development.
  - xcrunner529 2y ago
    I am hardly a dev but occasionally have had to do some or some scripting or web stuff and have really loved VSCode and using the remote SSH support to basically feel like I’m coding locally. Does that not work for your devs?
speedisavirus 2y ago
That's fine for some. However it's not always that. I wrote an entire site on my ipad in spare time with GitPods. Maybe you are at a small company with a small team so if things get critical you are likely to get a call. Do you say F'it, do you carry your laptop, or do you carry your ipad like you already are knowing you can still at least do triage if needed because you have a perfectly configured gitpod to use.
2075 2y ago
I think nowadays the value of source code is rarely a more valuable asset than the data being processed. Also I would prefer to give my devs just a second machine to run workloads and eventually pull in data or mock the data so they get moving more easily.
neilv 2y ago
> The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). ). I'm my experience, the vast majority of companies should worry less about this [...] I once had to burn a ton of political capital (including some on credit), because someone who didn't understand software thought that cutting-edge tech startup software developers, even including systems programmers working close to metal, could work effectively using only virtual remote desktops... with a terrible VM configuration... from servers literally halfway around the world... through a very dodgy firewall and VPN... of 10Mb/s total bandwidth... for the entire office of dozens of developers. (And no other Internet access from the VMs. Administrators would copy whatever files from the Internet that are needed for work. And there was a bureaucratic form for a human process, if you wanted to request any code/data to go in or out. And the laptops/workstations used only as thin-clients for the remote VMs would have to be Windows and run this ridiculous obscure 'endpoint security' software that had changed hands from its ancient developer, and hadn't even updated the marketing materials (e.g., a top bulletpoint was keeping your employees from wasting time on a Web site that famously was wiped out over a decade earlier), and presumably was littered with introduced vulnerabilities and instabilities.) Note that this was not something like DoD, nor HIPAA, nor finance. Just cutting-edge tech on which (ironically) we wanted first-mover advantage. This escalated to the other top-titled software engineer and I together doing a presentation to C-suite, on why not only would this kill working productivity (especially in a startup that needed to do creative work fast!), but the bad actors someone was paranoid about could easily circumvent it anyway to exfiltrate data (using methods obvious to the skilled software people like they hired, some undetectable by any security product or even human monitoring they imagined), and all the good rule-following people would quit in incredulous frustration. Unfortunately, it might not have been even the CEO's call, but a crazy investor.
hintymad 2y ago
> Personally - just let the developer own the machine they use for development. It'll work if the company can offer something similar to EC2. Unfortunately most of the companies are not capable of doing so if they are not on cloud.
jt2190 2y ago
I’m not sure we should leap from: > I have seen several attempts to move dev environments to a remote host. They invariably suck. To “therefore they will always suck and have no benefits and nobody should ever use them ever”. Apologies for the hyperbole but I’m making a point that comments like these tend to shut down interesting explorations of the state of the art of remote computing and what the pros/cons are. Edit: In a world where users demand that companies implement excellent security then we must allow those same companies to limit physical access to their machines as much as possible.
- horsawlarway 2y ago
  But they don't suck because of lack of effort - they suck because there are real physical constraints. Ex - even on a VERY good connection, RTT on the network is going to exceed your frame latency for a computer sitting in front of you (before we even get into the latency of the actual frame rendering of that remote computer). There's just not a solution for "make the light go faster". Then we get into the issues the author actually laid out quite compellingly - Shared resources are unpredictable. Is my code running slowly right now because I just introduced an issue, or is it because I'm sharing an env and my neighbor just ate 99% of the CPU/IO, or my network provider has picked a different route and my latency just went up 500ms? And that's before we even touch the "My machine is down/unreachable, I don't know why and I have no visibility into resolving the issue, when was my last commit again?" style problems... > Edit: In a world where users demand that companies implement excellent security then we must allow those same companies to limit physical access to their machines as much as possible. And this... is just bogus. We're not talking about machines running production data. We're talking about a developer environment. Sure - limit access to prod machines all you like, while you're at it, don't give me any production user data either - I sure as hell don't want it for local dev. What I do want is a fast system that I control so that I can actually tweak it as needed to develop and debug the system - it is almost impossible to give a developer "the least access needed" to do development locally because if you know what that access was you wouldn't be developing still.
  - sangnoir 2y ago
    > But they don't suck because of lack of effort - they suck because there are real physical constraints. They do suck due to lack of effort or investment. FANG companies have remote dev experiences that are decent - or even great - because they invest obscene amounts into dev tooling. There physical constraints on the flipside: especially for gigantic codebases or datasets that don't fit on dev laptops or have need lower latencies to other services in the DC. Added bonus: smaller attack surface area for adversaries who want to gain access to your code.
nixdev 2y ago
> Personally - just let the developer own the machine they use for development. Overall I agree with you that this is how it should be, but as DevOps working with so many development teams, I can tell you that too many developers know a language or two but beyond that barely know how to use a computer. Most developers (yes even most of the ones in Silicon Valley or the larger Bay Area) with Macbooks will smile and nod at when you tell them that Docker Desktop runs a virtual machine to run a copy of Linux to run oci images, and then not too much later reveal themselves to have been clueless. Commenters on this site are generally expected to be in a different category. Just wanted to share that, as a seasoned DevOps pro, I can tell you it's pretty rough out there.
- sgarland 2y ago
  This is an unpopular take, but entirely true. Skilled at a programming language, other than maybe C, does not in any way translate to general skill with system administration, or even knowing how to correctly operate a computer. I once had to explain to a dev that their Mac was out of disk space because a. They had never removed dangling containers or old image versions b. They had never emptied the Trash.
  - pxc 2y ago
    Unpopular take? I thought it was common knowledge that computer operation and computer programming involve distinct (if overlapping) skillsets.
    - nixdev 2y ago
      It's unpopular because developers want to have root access to their machines whilst being proudly ignorant to how fast the sensitive medical data or financial data they're working on can fly out of the machine. Even when provided a means to instantiate virtual machines where they can have root access within the virtual machine, a lot of them will bitch.
      - pxc 2y ago
pmarreck 2y ago
In my last role as a director of engineering at a startup, I found that a project `flake.nix` file (coupled with simply asking people to use https://determinate.systems/posts/determinate-nix-installer/ https://determinate.systems/posts/determinate-nix-installer/ to install Nix) led to the fastest "new-hire-to-able-to-contribute" time of anything I've seen. Unfortunately, after a few hires (hand-picked by me), this is what happened: 1) People didn't want to learn Nix, neither did they want to ask me how to make something work with Nix, neither did they tell me they didn't want to learn Nix. In essence, I told them to set the project up with it, which they'd do (and which would be successful, at least initially), but forgot that I also had to sell them on it. In one case, a developer spent all weekend (of HIS time) uninstalling Nix and making things work using the "usual crap" (as I would call it), all because of an issue I could have fixed in probably 5 minutes if he had just reached out to me (which he did not, to my chagrin). The first time I heard them comment their true feelings on it was when I pushed back regarding this because I would have gladly helped... I've mentioned this on various Slacks to get feedback and people have basically said "you either insist on it and say it's the only supported developer-environment-defining framework, or you will lose control over it" /shrug 2) Developers really like to have control over their own machines (but I failed to assume they'd also want this control over the project dependencies, since, after all, I was the one who decided to control mine with the flake.nix in the first place!) 3) At a startup, execution is everything and time is possibly too short (especially if you have kids) to learn new things that aren't simple, even if better... that unfortunately may include Nix. 4) Nix would also be perfect for deployments... except that there is no (to my knowledge) general-purpose, broadly-accepted way to deploy via Nix, except to convert it to a Docker image and deploy that, which (almost) defeats most of the purpose of Nix. I still believe in Nix but actually trying to use it to "perfectly control" a team's project dependencies (which I will insist it does do, pretty much, better than anything else) has been a mixed bag. And I will still insist that for every 5 minutes spent wrestling with Nix trying to get it to do what you need it to do, you are saving at least an order of magnitude more time spent debugging non-deterministic dependency issues that (as it turns out) were only "accidentally" working in the first place.
- oblio 2y ago
  I think if you take about 80% of your comment and replace "Nix" with "Haskell/Lisp" and a few other techs, you'd basically have the same thing. Especially point #1.
brunoborges 2y ago
> Personally - just let the developer own the machine they use for development. I wonder if Microsoft's approach for Dev Box is the right one.
- codethief 2y ago
  Could you elaborate on what that approach is?
to11mtm 2y ago
laughs in "Here's a VDI with 2vCPUs and 32GB of RAM but the cluster is overloaded, also you get to budget which IDEs you have installed because you have only a couple hundred GB of storage for everything including what we install on the base image that you will never use"
reissbaker 2y ago
If your app fits on one machine, I agree with you: you absolutely should not use cloud dev environments in my opinion (and I've worked on large dev infra teams, that shipped cloud dev environments). The performance and latency of a Macbook Pro (or Framework 13, or whatever) is going to destroy cloud perf for development purposes. If it doesn't fit on one machine, though, you don't have another option: Meta, for example, will never have a local dev env for Instagram or Blue. Then you need to make some hard choices. Personally, my ideal cloud dev env is: 1. Local checkout of the code you're working on. You can use whatever IDE or text editor you prefer. For large monorepos, you'll need some special tooling to make sure it's easy to only check out slices of the repo. 2. Sync the code to the remote execution environment automatically, with hot-reloading. 3. Auto-port-forward from your local machine to the remote. 4. Optionally be able to run dependent services on your personal remote to debug/test their interactions with each other, and optionally be able to connect to a well-maintained shared environment for dependencies you aren't working on. If you have a shared environment, it can't be viewed as less-important than production: if it's broken, it's a SEV and the team that broke it needs to drop everything and fix it immediately. (Otherwise the shared env will be broken all the time, and your shipping speed will either drop, or you'll constantly be shipping bugs to prod due to lack of dev care.) At Meta we didn't have (1): everyone had to use VSCode, with special in-house plugins that synced to the remote environment. It was okay but honestly a little soul-sucking; I think customizing your tooling is part of a lot of people's craft and helps maintain their flow state. Thankfully we had the rest, so it was tolerable if not enjoyable. At Airbnb we didn't have the political will to enforce (4), so the dev env was always broken. I think (4) is actually the most critical part: it doesn't matter how good the rest of it is, if the org doesn't care about it working. But yeah — if you don't need it, that's a lot of work and politics. Use local environments as long as you possibly can.
czhu12 2y ago
We tried this approach at a former company with ~600 engineers at the time. Trying to boot the full service on a single machine required every single developer in the company installing ~50ish microservices on their machine, for things to work correctly. Became totally intractable. I guess one can grumble about bad architecture all day but this had to be solved. we had to move to remote development environments which restored everyone’s sanity. Both FAANG companies I’ve worked at had remote dev environments that were built in house.
- rileymat2 2y ago
  I may be misunderstanding, but wouldn't you want the particular microservice you are working on independent enough to develop locally, then deploy into the remote environment to test the integration? (I don't work at this scale)
  - animal_spirits 2y ago
    Invariably this is an ideal and does not match up in reality. I work at ~50 ish employee company and we have layers of dependencies between at least 6 or 7 various microservices. I can see this adding up in complexity as the product scales
    - rileymat2 2y ago
      Yes, the real world trumps theory, hence my question.
      - jrs235 2y ago
        Reminds me of a favorite quote: "What's the difference between between theory and practice (reality)? In theory, they're the same."
    - flimsypremise 2y ago
      If the microservice has dependencies on other services it is not a microservice.
cryptonector 2y ago
I've no problem with remote dev envs for most things. But they have to be VMs in many cases, not containers.
peeters 2y ago
> The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). I've worked in a remote, secured development environment and it sucked, but to their credit the company did it for exactly this reason - control over the source. But bear in mind that source control is a two-way street. Losing proprietary source can be harmful (especially in compiled languages where the source might carry much more information than the distributable). But they were mostly worried about the opposite way...that something malicious gets INTO the source which could pose an existential threat. You'd be correct to say "well that should be the domain of source control, peer review etc", but in this case the company assessed the risk high enough to do both.
fensgrim 2y ago
> how a developer manages assets like source code IMO there are some workloads, where it is beneficial for a developer to have access to a local repository with at least some snippets based on previous projects. Having a leftover PoC of some concept written for a previous employer but never elevated to team use/production is both handy (at least to confirm that the build environment is still viable after an unspecified period of toolchain updates) and ethical (copying production code is not ethical - even if the old and new products are vastly different e.g. last job was taxi app, new app is banking app). Making it all 'remote' and 'cloud' will eventually result in a bike reinvention penalty on each new employment - not everything can be rebuilt from memory only, especially things that are done 1-2 times a year; sure there is open-source documentation/examples, but at some point it'll just introduce even heavier penalty for a need to either know a lot of opensource stuff to have some reference points, or to work on a pet projects to get the same amount of references.
- scarface_74 2y ago
  Are you suggesting that you should enable the employee to move work done on company time and that is the company’s IP to a new company? And the new company would also be liable for using trade secrets that they shouldn’t.
  - fensgrim 2y ago
    Neither, it's unethical and there's no possibility of doing that in legal way. However I do write 1-2 hour PoCs on my spare time and my own equipment, using only publicly available stuff - they sometimes come handy at some point later. If we assume 'remote first' development is okay - with no possibility to test stuff locally, well, we're back to either bookmark managers or pet projects to keep at least a bit of knowledge between jobs.

dangsux 2y ago

[dead]

shriek 2y ago

And the reason they suck is the feedback loop is just too high as compared to running it locally. You have to jump through hoops to debug/troubleshoot your code or any issues that you come across between your code and output of your code. And it's almost impossible to work on things when you have spotty internet. I haven't worked on extremely sensitive data but for PII data from prod to dev, scrubbing is a good practice to follow. This will vary based on the project/team you're on of course.

ethbr1 2y ago
Aka 'if a developer knew beforehand everything they needed, it wouldn't be development'
marcosdumay 2y ago
That's the least important problem. The developers also lack knowledge about the environment; can't evolve the environment; can't test the environment for bugs; and invariably interfere with each other because it's never isolated well. And also, yes, it adds lag. Anyway, yes, working locally on false data that little resemblance to production still beats remote environments.

binary132 2y ago

Hello. Currently debugging my kubernetes-based dev pod and not getting anything else done. What fun!

deleted 2y ago

[deleted]

haolez 2y ago

Sometimes I don't even use virtual envs when developing locally in Python. I just install everything that I need with pip --user and be done with it. Never had any conflicts with system packages whatsoever. If I somehow break my --user environment, I simply delete it and start again. Never had any major version mismatch in dependencies between my machine and what was running in production. At least not anything that would impact the actual task that I was working on. I'm not recommending this as a best practice. I just believe that we, as developers, end up creating some myths to ourselves of what works and what doesn't. It's good to re-evaluate these beliefs now and then.

__MatrixMan__ 2y ago
When doing this re-evaluation, please consider that others might be quietly working very hard to discover and recreate locally whatever secret sauce you and production share.
ctippett 2y ago
I'm not going to second-guess what works for you, but Python makes it so easy to work with an ephemeral environment. python -m venv .venv
- haolez 2y ago
  Yeah, I know. But then you have to make sure that your IDE is using the correct environment, that the notebook is using the correct environment, that the debugger is using the correct environment. It's trivial to setup a venv, but sometimes it's just not worth it for me.
  - zo1 2y ago
    This is one of the main reasons I tell people not to use VSCode. The people most likely to use it are juniors and people new to python specifically, and they're the most likely to fall victim to 'but my "IDE" says it's running 3.8 with everything installed, but when I run it from my terminal it's a different python 3.8' I watched it last week. With 4 (I hope junior) Devs in a "pair programming" session that forced me to figure out how VSCode does virtual envs, and still I had to tell them like 3 times "stop opening a damn new terminal, it's obviously not setup with our python version, run the command inside the one that has the virtual env activated".

csweichel 2y ago

OP here. There definitely is a place for running things on your local machine. Exactly as you say: one can get a great deal of consistency using VMs. One of the benefits of moving away from Kubernetes, to a runner-based architecture , is that we can now seamlessly support cloud-based and local environments (https://www.gitpod.io/blog/introducing-gitpod-desktop https://www.gitpod.io/blog/introducing-gitpod-desktop). What's really nice about this is that with this kind of integration there's very little difference in setting up a dev env in the cloud or locally. The behaviour and qualities of those environments can differ vastly though (network bandwidth, latency, GPU, RAM, CPUs, ARM/x86).

edplmajor 2y ago
Need middleware libs that react to eBPF data and signal app code to scale up/down forks in their own memory VM, like V8 Kubernetes is another mess of userspace ops tools. Userspace is for composable UI not backend. Kube and Chef and all those other ops tools are backend functionality being used like UI by leet haxxors
master_crab 2y ago
Hi Christian. We just deployed Gitpod EKS at our company in NY. Can we get some details on the replacement architecture? I’m sure it’s great but the devil is always in the details.
michaelt 2y ago
> The behaviour and qualities of those environments can differ vastly though (network bandwidth, latency, GPU, RAM, CPUs, ARM/x86). For example, when you're running on your local machine you've actually got the amount of RAM and CPU advertised :)
- sethammons 2y ago
  "Hm, why does my Go service on a pod with 2.2 cpu's think it has 6k? Oh, it thinks it has the whole cluster. Nice; that is why scheduling has been an issue"
justinclift 2y ago

lotharcable 2y ago

I strongly recommend just switching the Dev environment over to Linux and taking advantage of tools like "distrobox" and "toolbx". https://github.com/89luca89/distrobox https://github.com/89luca89/distrobox https://containertoolbx.org/ https://containertoolbx.org/ It is sorta like Vagrant, but instead of using virtualbox virtual machines you use podman containers. This way you get to use OCI images for your "dev environment" that integrates directly into your desktop. https://podman.io/ https://podman.io/ There is some challenges related to usermode networking for non-root-managed controllers and desktop integration has some additional complications. But besides that it has almost no overhead and you can have unfettered access to things like GPUs. Also it is usually pretty easy to convert your normal docker or kubernetes containers over to something you can run on your desktop. Also it is possible to use things like Kubernetes pods definitions to deploy sets of containers with podman and manage it with systemd and such things. So you can have "clouds of containers" that your dev container needs access to locally. If there is a corporate need for window-specific applications then running Windows VMs or doing remote applications over RDP is a possible work around. If everything you are targeting as a deployment is going to be Linux-everything then it doesn't make a lot of sense to jump through a bunch of hoops and cause a bunch of headaches just to avoid having it as workstation OS.

trog 2y ago
If you're doing this, there are many cases where you might as well just spin up a decent Linux server and give your developers accounts on that? With some pretty basic setup everyone can just run their own stuff within their own user account. You'll run into occasional issues (e.g. if everyone is trying to run default node.js on default port) but with some basic guardrails it feels like it should be OK? I'm remembering back to when my old company ran a lot of PHP projects. Each user just had their own development environment and their own Apache vhost. They wrote their code and tested it in their own vhost. Then we'd merge to a single separate vhost for further testing. I am trying to remember anything about what was painful about it but it all basically Just Worked. Everyone had remote access via VPN; the worst case scenario for them was they'd have to work from home with a bit of extra latency.
- 2y ago

idunnoman1222 2y ago

Sounds like you are not using a lot of hardware - Rfid, POS, top-spec video cards, etc

0xbadcafebee 2y ago

Most teams/products I have been involved in, the stack always grows to the point that a dev can no longer test it on their own machine, regardless of how big the machine is. And having a different development machine than production leads to completely predictable and unavoidable problems. Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing.

hosh 2y ago
I have used remote dev machines just fine, but my workflow vastly differs from many of my coworkers: terminal-only spacemacs + tmux + mosh. I have a lot of CLI and TUI tools, and I do not use VScode at all. The main GUI app I run is a browser, and that runs locally. I have worked on developing VMs for other developers that rely on a local IDE such. The main sticking point is syncing and schlepping source code (something my setup avoids because the source code and editor is on the remote machine). I have tried a number of approaches, and I sympathize with the article author. So, in response to "Devs need to create the software tooling to make remote dev less painful. I mean, they're devs... making software is kind of their whole thing." <-- syncing and schlepping source code is by no means a solved problem. I can also say that, my spacemacs config is very vanilla. Like my phone, I don't want to be messing with it when I want to code. Writing tooling for my editor environment is a sideshow for the work I am trying to finish.
- pjmlp 2y ago
  Me as well, specially in the days that there was only a UNIX dev server for everyone. It was never an issue to use X Windows on them, with hummingbird on my Windows thin client. I guess a new generation has to learn the ways of timesharing development.
- xcrunner529 2y ago
  I am hardly a dev but occasionally have had to do some or some scripting or web stuff and have really loved VSCode and using the remote SSH support to basically feel like I’m coding locally. Does that not work for your devs?

speedisavirus 2y ago

That's fine for some. However it's not always that. I wrote an entire site on my ipad in spare time with GitPods. Maybe you are at a small company with a small team so if things get critical you are likely to get a call. Do you say F'it, do you carry your laptop, or do you carry your ipad like you already are knowing you can still at least do triage if needed because you have a perfectly configured gitpod to use.

2075 2y ago

I think nowadays the value of source code is rarely a more valuable asset than the data being processed. Also I would prefer to give my devs just a second machine to run workloads and eventually pull in data or mock the data so they get moving more easily.

neilv 2y ago

> The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). ). I'm my experience, the vast majority of companies should worry less about this [...] I once had to burn a ton of political capital (including some on credit), because someone who didn't understand software thought that cutting-edge tech startup software developers, even including systems programmers working close to metal, could work effectively using only virtual remote desktops... with a terrible VM configuration... from servers literally halfway around the world... through a very dodgy firewall and VPN... of 10Mb/s total bandwidth... for the entire office of dozens of developers. (And no other Internet access from the VMs. Administrators would copy whatever files from the Internet that are needed for work. And there was a bureaucratic form for a human process, if you wanted to request any code/data to go in or out. And the laptops/workstations used only as thin-clients for the remote VMs would have to be Windows and run this ridiculous obscure 'endpoint security' software that had changed hands from its ancient developer, and hadn't even updated the marketing materials (e.g., a top bulletpoint was keeping your employees from wasting time on a Web site that famously was wiped out over a decade earlier), and presumably was littered with introduced vulnerabilities and instabilities.) Note that this was not something like DoD, nor HIPAA, nor finance. Just cutting-edge tech on which (ironically) we wanted first-mover advantage. This escalated to the other top-titled software engineer and I together doing a presentation to C-suite, on why not only would this kill working productivity (especially in a startup that needed to do creative work fast!), but the bad actors someone was paranoid about could easily circumvent it anyway to exfiltrate data (using methods obvious to the skilled software people like they hired, some undetectable by any security product or even human monitoring they imagined), and all the good rule-following people would quit in incredulous frustration. Unfortunately, it might not have been even the CEO's call, but a crazy investor.

hintymad 2y ago

> Personally - just let the developer own the machine they use for development. It'll work if the company can offer something similar to EC2. Unfortunately most of the companies are not capable of doing so if they are not on cloud.

jt2190 2y ago

I’m not sure we should leap from: > I have seen several attempts to move dev environments to a remote host. They invariably suck. To “therefore they will always suck and have no benefits and nobody should ever use them ever”. Apologies for the hyperbole but I’m making a point that comments like these tend to shut down interesting explorations of the state of the art of remote computing and what the pros/cons are. Edit: In a world where users demand that companies implement excellent security then we must allow those same companies to limit physical access to their machines as much as possible.

horsawlarway 2y ago
But they don't suck because of lack of effort - they suck because there are real physical constraints. Ex - even on a VERY good connection, RTT on the network is going to exceed your frame latency for a computer sitting in front of you (before we even get into the latency of the actual frame rendering of that remote computer). There's just not a solution for "make the light go faster". Then we get into the issues the author actually laid out quite compellingly - Shared resources are unpredictable. Is my code running slowly right now because I just introduced an issue, or is it because I'm sharing an env and my neighbor just ate 99% of the CPU/IO, or my network provider has picked a different route and my latency just went up 500ms? And that's before we even touch the "My machine is down/unreachable, I don't know why and I have no visibility into resolving the issue, when was my last commit again?" style problems... > Edit: In a world where users demand that companies implement excellent security then we must allow those same companies to limit physical access to their machines as much as possible. And this... is just bogus. We're not talking about machines running production data. We're talking about a developer environment. Sure - limit access to prod machines all you like, while you're at it, don't give me any production user data either - I sure as hell don't want it for local dev. What I do want is a fast system that I control so that I can actually tweak it as needed to develop and debug the system - it is almost impossible to give a developer "the least access needed" to do development locally because if you know what that access was you wouldn't be developing still.
- sangnoir 2y ago
  > But they don't suck because of lack of effort - they suck because there are real physical constraints. They do suck due to lack of effort or investment. FANG companies have remote dev experiences that are decent - or even great - because they invest obscene amounts into dev tooling. There physical constraints on the flipside: especially for gigantic codebases or datasets that don't fit on dev laptops or have need lower latencies to other services in the DC. Added bonus: smaller attack surface area for adversaries who want to gain access to your code.

nixdev 2y ago

> Personally - just let the developer own the machine they use for development. Overall I agree with you that this is how it should be, but as DevOps working with so many development teams, I can tell you that too many developers know a language or two but beyond that barely know how to use a computer. Most developers (yes even most of the ones in Silicon Valley or the larger Bay Area) with Macbooks will smile and nod at when you tell them that Docker Desktop runs a virtual machine to run a copy of Linux to run oci images, and then not too much later reveal themselves to have been clueless. Commenters on this site are generally expected to be in a different category. Just wanted to share that, as a seasoned DevOps pro, I can tell you it's pretty rough out there.

sgarland 2y ago
This is an unpopular take, but entirely true. Skilled at a programming language, other than maybe C, does not in any way translate to general skill with system administration, or even knowing how to correctly operate a computer. I once had to explain to a dev that their Mac was out of disk space because a. They had never removed dangling containers or old image versions b. They had never emptied the Trash.
- pxc 2y ago
  Unpopular take? I thought it was common knowledge that computer operation and computer programming involve distinct (if overlapping) skillsets.
  - nixdev 2y ago
    It's unpopular because developers want to have root access to their machines whilst being proudly ignorant to how fast the sensitive medical data or financial data they're working on can fly out of the machine. Even when provided a means to instantiate virtual machines where they can have root access within the virtual machine, a lot of them will bitch.
    - pxc 2y ago

pmarreck 2y ago

In my last role as a director of engineering at a startup, I found that a project `flake.nix` file (coupled with simply asking people to use https://determinate.systems/posts/determinate-nix-installer/ https://determinate.systems/posts/determinate-nix-installer/ to install Nix) led to the fastest "new-hire-to-able-to-contribute" time of anything I've seen. Unfortunately, after a few hires (hand-picked by me), this is what happened: 1) People didn't want to learn Nix, neither did they want to ask me how to make something work with Nix, neither did they tell me they didn't want to learn Nix. In essence, I told them to set the project up with it, which they'd do (and which would be successful, at least initially), but forgot that I also had to sell them on it. In one case, a developer spent all weekend (of HIS time) uninstalling Nix and making things work using the "usual crap" (as I would call it), all because of an issue I could have fixed in probably 5 minutes if he had just reached out to me (which he did not, to my chagrin). The first time I heard them comment their true feelings on it was when I pushed back regarding this because I would have gladly helped... I've mentioned this on various Slacks to get feedback and people have basically said "you either insist on it and say it's the only supported developer-environment-defining framework, or you will lose control over it" /shrug 2) Developers really like to have control over their own machines (but I failed to assume they'd also want this control over the project dependencies, since, after all, I was the one who decided to control mine with the flake.nix in the first place!) 3) At a startup, execution is everything and time is possibly too short (especially if you have kids) to learn new things that aren't simple, even if better... that unfortunately may include Nix. 4) Nix would also be perfect for deployments... except that there is no (to my knowledge) general-purpose, broadly-accepted way to deploy via Nix, except to convert it to a Docker image and deploy that, which (almost) defeats most of the purpose of Nix. I still believe in Nix but actually trying to use it to "perfectly control" a team's project dependencies (which I will insist it does do, pretty much, better than anything else) has been a mixed bag. And I will still insist that for every 5 minutes spent wrestling with Nix trying to get it to do what you need it to do, you are saving at least an order of magnitude more time spent debugging non-deterministic dependency issues that (as it turns out) were only "accidentally" working in the first place.

oblio 2y ago
I think if you take about 80% of your comment and replace "Nix" with "Haskell/Lisp" and a few other techs, you'd basically have the same thing. Especially point #1.

brunoborges 2y ago

> Personally - just let the developer own the machine they use for development. I wonder if Microsoft's approach for Dev Box is the right one.

codethief 2y ago
Could you elaborate on what that approach is?

to11mtm 2y ago

laughs in "Here's a VDI with 2vCPUs and 32GB of RAM but the cluster is overloaded, also you get to budget which IDEs you have installed because you have only a couple hundred GB of storage for everything including what we install on the base image that you will never use"

reissbaker 2y ago

If your app fits on one machine, I agree with you: you absolutely should not use cloud dev environments in my opinion (and I've worked on large dev infra teams, that shipped cloud dev environments). The performance and latency of a Macbook Pro (or Framework 13, or whatever) is going to destroy cloud perf for development purposes. If it doesn't fit on one machine, though, you don't have another option: Meta, for example, will never have a local dev env for Instagram or Blue. Then you need to make some hard choices. Personally, my ideal cloud dev env is: 1. Local checkout of the code you're working on. You can use whatever IDE or text editor you prefer. For large monorepos, you'll need some special tooling to make sure it's easy to only check out slices of the repo. 2. Sync the code to the remote execution environment automatically, with hot-reloading. 3. Auto-port-forward from your local machine to the remote. 4. Optionally be able to run dependent services on your personal remote to debug/test their interactions with each other, and optionally be able to connect to a well-maintained shared environment for dependencies you aren't working on. If you have a shared environment, it can't be viewed as less-important than production: if it's broken, it's a SEV and the team that broke it needs to drop everything and fix it immediately. (Otherwise the shared env will be broken all the time, and your shipping speed will either drop, or you'll constantly be shipping bugs to prod due to lack of dev care.) At Meta we didn't have (1): everyone had to use VSCode, with special in-house plugins that synced to the remote environment. It was okay but honestly a little soul-sucking; I think customizing your tooling is part of a lot of people's craft and helps maintain their flow state. Thankfully we had the rest, so it was tolerable if not enjoyable. At Airbnb we didn't have the political will to enforce (4), so the dev env was always broken. I think (4) is actually the most critical part: it doesn't matter how good the rest of it is, if the org doesn't care about it working. But yeah — if you don't need it, that's a lot of work and politics. Use local environments as long as you possibly can.

czhu12 2y ago

We tried this approach at a former company with ~600 engineers at the time. Trying to boot the full service on a single machine required every single developer in the company installing ~50ish microservices on their machine, for things to work correctly. Became totally intractable. I guess one can grumble about bad architecture all day but this had to be solved. we had to move to remote development environments which restored everyone’s sanity. Both FAANG companies I’ve worked at had remote dev environments that were built in house.

rileymat2 2y ago
I may be misunderstanding, but wouldn't you want the particular microservice you are working on independent enough to develop locally, then deploy into the remote environment to test the integration? (I don't work at this scale)
- animal_spirits 2y ago
  Invariably this is an ideal and does not match up in reality. I work at ~50 ish employee company and we have layers of dependencies between at least 6 or 7 various microservices. I can see this adding up in complexity as the product scales
  - rileymat2 2y ago
    Yes, the real world trumps theory, hence my question.
    - jrs235 2y ago
      Reminds me of a favorite quote: "What's the difference between between theory and practice (reality)? In theory, they're the same."
  - flimsypremise 2y ago
    If the microservice has dependencies on other services it is not a microservice.

cryptonector 2y ago

I've no problem with remote dev envs for most things. But they have to be VMs in many cases, not containers.

peeters 2y ago

> The only real downside is data control (ie - the company has less control over how a developer manages assets like source code). I've worked in a remote, secured development environment and it sucked, but to their credit the company did it for exactly this reason - control over the source. But bear in mind that source control is a two-way street. Losing proprietary source can be harmful (especially in compiled languages where the source might carry much more information than the distributable). But they were mostly worried about the opposite way...that something malicious gets INTO the source which could pose an existential threat. You'd be correct to say "well that should be the domain of source control, peer review etc", but in this case the company assessed the risk high enough to do both.

fensgrim 2y ago

> how a developer manages assets like source code IMO there are some workloads, where it is beneficial for a developer to have access to a local repository with at least some snippets based on previous projects. Having a leftover PoC of some concept written for a previous employer but never elevated to team use/production is both handy (at least to confirm that the build environment is still viable after an unspecified period of toolchain updates) and ethical (copying production code is not ethical - even if the old and new products are vastly different e.g. last job was taxi app, new app is banking app). Making it all 'remote' and 'cloud' will eventually result in a bike reinvention penalty on each new employment - not everything can be rebuilt from memory only, especially things that are done 1-2 times a year; sure there is open-source documentation/examples, but at some point it'll just introduce even heavier penalty for a need to either know a lot of opensource stuff to have some reference points, or to work on a pet projects to get the same amount of references.

scarface_74 2y ago
Are you suggesting that you should enable the employee to move work done on company time and that is the company’s IP to a new company? And the new company would also be liable for using trade secrets that they shouldn’t.
- fensgrim 2y ago
  Neither, it's unethical and there's no possibility of doing that in legal way. However I do write 1-2 hour PoCs on my spare time and my own equipment, using only publicly available stuff - they sometimes come handy at some point later. If we assume 'remote first' development is okay - with no possibility to test stuff locally, well, we're back to either bookmark managers or pet projects to keep at least a bit of knowledge between jobs.