19 ms·

Devpod: Remote development environment at Uber

291 points by phgn 4y ago

mmmpetrichor 4y ago
TLDR for vscode remote users? Let me be lazy and just get a Y/N over whether it has any benefit over the normal remote vscode UX.
- lozenge 4y ago
  They reimplemented GitHub Codespaces without GitHub
trynewideas 4y ago
Very curious how, or if, this helps or affects non-developers also working in the monorepo, like tech writers, QA, UX/UI/visual designers, training/enablement, support, etc.
cloudking 4y ago
This sounds awesome, but not practical for smaller companies. The cloud costs must be significant? "Faster Git, Build, and IDE experience by utilizing beefy cloud resources–up to 48 cores and 96 GB of RAM per environment"
- jeffbee 4y ago
  Why would it need to be per user? An environment of that size could easily support many users. Developers spend most of their time just staring and only intermittently need to build. There is no particular reason why everyone would be building at once. Most of the time you'll get all those cores to yourself even though there could be dozens of devs sharing the box.
  - gabereiser 4y ago
    We’re coming around full circle to the old days of thin client terminals to a beefy mainframe under the corporation’s control. Only it’s hosted in the cloud and all your data is at risk.
    - dilyevsky 4y ago
      You can run it not in cloud. Uber uses its own DCs (although not sure they do that for dev). It’ll probably be more expensive because you can’t scale everything down for the night (unless you have dev teams spread globally).
  - epolanski 4y ago
    Also, smaller company can probably go a long way just on codespaces. If GitHub can, 99% of projects out there likely can too.
- cyc116 4y ago
  Does smaller companies have mono repos that size?
- Moto7451 4y ago
  Correct. My last job built a low rent version of this and it was very painful. 15-45 minutes to launch a pod that would have taken 3 minutes under Compose. Pushed a new base image? Whole thing would roll over and need another 15-45 minutes. To do this well it takes manpower. There are some off the shelf options out there for remote/prod like Kubernetes based development that are smarter choices than building your own. The table stakes for doing this is being a big corp that can afford to spend millions on one custom tool with a hope for positive ROI. For the rest of us, find something that works that is open source or easily licensed.
deleted 4y ago
[deleted]
ValleZ 4y ago
Most developers in Uber use local builds anyway because of input lag. Dev pods were mostly good to prevent laptop from working at 100% fan volume at all times though.
- mrhands556 4y ago
  input lag like when typing? I use remote env on vscode and afaik it’s locally editing and syncing after the fact for lint, compilation etc
  - judge2020 4y ago
    It was probably just a RDP connection, which either signals how old this insight is, or how far behind Uber is on these new ways to remotely code.
    - esprehn 4y ago
      The post clearly shows they're not using RDP. That said Projector round tripped all the key strokes to get new draw commands (unlike VSCode) which also results in lag. The new IntelliJ remote architecture is much better, and it seems Uber is moving that way too.
      - judge2020 4y ago
        They're using this now, but I imagine the user ValleZ's comment is about a past time working at Uber or hearing stories from Uber developer friends, in which I can see why spinning up a low-latency (but still over 30+ millisecond RTT) virtual machine would be an early solution to developers trying to cut down on build times and whatnot.
      - Noumenon72 4y ago
        What's the new way? JetBrains Gateway? Have they caught up to VSCode's devcontainers?
- WookieRushing 4y ago
prakashn27 4y ago
We already have something like this at Meta for improving developer experience. All our dev environments are remote
- fragmede 4y ago
  Any externally visible docs on it? Google's is documented at https://cloud.google.com/blog/topics/developers-practitioners/googles-virtual-desktop-future https://cloud.google.com/blog/topics/developers-practitioner...
  - nsenifty 4y ago
    I don't think it's been discussed externally yet. It's called on-demand (some references to it here - https://developers.facebook.com/blog/post/2022/11/15/meta-developers-workflow-exploring-tools-used-to-code/ https://developers.facebook.com/blog/post/2022/11/15/meta-de...).
    - fragmede 4y ago
      Fascinating, thanks!
xnx 4y ago
Similar setup at Google: https://cloud.google.com/blog/topics/developers-practitioners/googles-virtual-desktop-future https://cloud.google.com/blog/topics/developers-practitioner...
- jeffbee 4y ago
  The contrast between the Uber and Google setups would be twofold. Google engineers weren't doing local builds anyway (unless they intentionally inflicted those upon themselves) because of Forge. The Forge builders already had 10000 CPU cores in 2010[1]. You can probably imagine how many cores Forge or its equivalent has today. Secondly Google doesn't suffer from the performance problems of a weird freeware VCS, because they don't use git or the git workflow model at all. They built their own VCS based on the workflow model from Perforce. One funny aspect of Google moving development into cloud machines was that a decade before the process ran the other way. The desktop that was issued to most engineers was a production websearch machine turned sideways and stuck under your desk. 1: https://youtu.be/b52aXZ2yi08?t=1118 https://youtu.be/b52aXZ2yi08?t=1118
  - ridiculous_fish 4y ago
    Google contains multitudes. Android, Chrome, and ChromeOS are examples of large Google projects that do not use Perforce and are routinely built locally.
    - hoten 4y ago
      Local-ish. (at least) Chromium is built by Googlers w/ the aide of goma, basically a distcc[1]. [1] https://chromium.googlesource.com/chromium/src/+/778a7e84f65fd36658f91881bda07b9153a8729d/docs/linux_faster_builds.md#use-goma https://chromium.googlesource.com/chromium/src/+/778a7e84f65...
    - ameliaquining 4y ago
      Chrome, at least, is typically built in the cloud (I don't know about Android, though it would be weird for it to be the only major project that doesn't do this). It's not just that you can have beefier machines in the cloud (though that's a factor); it's also that you can have a shared cache of build artifacts for all your users, which makes most workflows a lot faster. See documentation: https://chromium.googlesource.com/infra/goma/client/ https://chromium.googlesource.com/infra/goma/client/
deleted 4y ago
[deleted]
Gigachad 4y ago
You will develop in the pod. You will not own a capable laptop.
- headbansown 4y ago
  You will fix the bugs?
teeray 4y ago
This sounds great. Certainly the IP zealots love it because the code never leaves the walled garden, and it’s a super-secure blah blah environment. Consistent tooling, easy onboarding… you get the idea. All good until you have an outage. Then your development team’s productivity drops to exactly zero while you fix it and your entire production environment is now potentially vulnerable to defects you can’t fix until the development environment is fixed (better hope it stays running). This is when companies realize that the development environment is actually a service and it needs higher SLA targets than production, but it will never get the attention that it needs to achieve those targets (because it’s just dev, right?).
- gandalfgeek 4y ago
  Every service has the risk of introducing a new point of failure, of course. But compare dev productivity lost due to service downtime with that of each new SWE in your org burning time to A) setup their own unique snowflake of an environment and B) futzing and debugging it when it breaks or there's a software update.
  - erik_seaberg 4y ago
    > burning time to setup their own unique snowflake of an environment Anyone good is doing that even if you tell him not to. One size does not fit all.
- charcircuit 4y ago
  You already have the same concern with source control, code review, continuous integration, continuous deployment, etc. Dev servers are arguably much simpler to provide without issues.
  - teeray 4y ago
    But I don’t really have those concerns—they’re better mitigated. If source control goes down, I have a complete clone on my machine and so do my coworkers (yay git). I have done code reviews by pasting `git am` formatted patches between coworkers during particularly long GitHub outages. CI/CD is more problematic, but I only need those things to integrate work. I can still do work and verify correctness with tests that I can run locally. None of that is possible in a centrally-managed dev VM setup. When the VMs go down, you send your dev team home until it’s fixed. You’re still paying their salaries while you pay another team to make them productive again.
gandalfgeek 4y ago
Back in the early 2000s, Sun ("the network is the computer") had a similar solution that worked seamlessly for most of their software org-- the Sun Ray. https://en.wikipedia.org/wiki/Sun_Ray https://en.wikipedia.org/wiki/Sun_Ray It was a network terminal. Your files and entire session were on the server. Your “local” terminal consisted only of a network interface and enough compute power to display your session. The way they had it set up was that you could insert your Sun employee ID – the same card used to get into the building – into a slot in the terminal. That authenticated you to the server and displayed your session instantly. Want to show a colleague something you’re working on? Just put your ID into their Sun Ray and show them exactly what you were doing. That was cool! It was a frictionless way to demo and collaborate.
- jimmaswell 4y ago
  What's old is new again. How soon until we realize the X window system actually had some good ideas again and start running desktop apps on cloud servers for remote work?
  - erik_seaberg 4y ago
    X maintainers are promoting a new platform that doesn’t provide for remote hardware rendering. Ironically, shipping Javascript to a web browser (local to the display) seems to be the way ahead for performance.
  - paxys 4y ago
    It had some good ideas, but that was the extent of it. Every iteration of the solution since then has agreed that streaming all application UI is always going to be janky and needless. It's a lot better to host the data on the server side and let the client handle visual rendering by itself.
    - jimmaswell 4y ago
      I used to run Firefox on my Debian server with X tunneling in high school sometimes. It wasn't that bad then and surely it's even better with today's internet.
EastSmith 4y ago
We are moving to such cloud environment, and it makes me sick. Maybe you need to dockerise Mongo, MySql and 5 other dependencies - I can get this, but I don't get it why the rest the code should still be running in the cloud. Python, Rails, Node? Why? Developers should be able to run 1 shell commands to install node. Dev experience excuses are just excuses for a bad setup. So, fix your setup, please. Not being able to run the project natively is big red flag for me and when I move jobs will be my 1st question.
- why5s 4y ago
  For work-related stuff: I disagree. We're not even at Uber's scale at my company and I hate having to manage the ever-changing set of dependencies that I have no control over.
  - layer8 4y ago
    The thing is, it doesn’t have to be ever-changing, and creating a reproducible working setup locally can be achieved by appropriate setup scripts.
    - paxys 4y ago
      Easy in theory, but always breaks down for a non-trivial project with >5 developers on it.
    - tylerhou 4y ago
      Setup scripts that mutate an operating system are fragile. Break something? Unless you understand how the scripts work (which will become stupidly complex at a scale like Uber’s) you will have to reinstall your machine. Or you will have to staff a ton of support to help users when they do break their configuration. Spinning up a VM with an image containing all the development tools is a much smoother experience most of the time. The only reason why I don’t use it where I work is because I use vim and network adds too much latency for me.
      - 4y ago
faizshah 4y ago
I like this workflow, use something like mutagen to sync from your local environment to the remote machine. For me, I prefer jetbrains IDEs so I tried using jetbrains gateway, it’s really buggy and slow. I prefer to use jetbrains ides on my local machine and use VSCode remote for interacting with the remote machine. I wonder though, is it worth it to setup a remote dev machine if you have an M1 Mac? I think you can get good compile times for Rust etc. on an M1. For me I have an intel Mac at work and at home so the remote dev env is better for builds.
- hokumguru 4y ago
  I think there are still some limits applied even on cutting edge hardware. GitHub previously said they hit network limitations cloning their multi-gb repo for instance.
  - mejutoco 4y ago
    If I am not mistaken, git clone is an atomic operation (it either succeeds or fails). Since I see it mentioned in many threads here: for a huge repo one can always use `git clone --depth 1` to get that repo and later do a proper pull to retrieve all history.
- nithayakumar 4y ago
  Yeah - we've (usenimbus.com - a company in this space) have heard the same about Jetbrains IDEs. You should check out our extension (or others') because there's been a lot of work put into making performance better. M1s are a mixed bag for this way of working. Pre-M1, devs were running into local computing power issues. Post-M1, more compatibility and stability issues.
quechimba 4y ago
For some reason this link redirects to https://www.uber.com/es-CO/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/es-CO/blog/devpod-improving-developer-p... which results in a 404. This URL works without redirects: https://www.uber.com/en-US/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/en-US/blog/devpod-improving-developer-p...
emptysea 4y ago
Wonder how it handles things like dot files? Also does it automatically install dependencies? Feel like running npm install each time (or the equivalent) would be slow.
chazeon 4y ago
Glad to see industry adopting cloud server + ssh + code-server / code-remote as a standard. As a PhD student I have been working on HPCs like this with an environment I built myself for a while. It is really good and helps you to move fast.
- icedchai 4y ago
  It depends on latency. If you have a slow network connection (or are simply far away from the cloud data center) you are in for pain. That being said, running a local environment on a constantly throttling 2019 MacBook Pro that sounds like a jet engine is probably worse.
eddsh1994 4y ago
This sounds very similar to my first IT Support role setting up thin clients for law firms in London that had VM instances for each role including development, and then a networked file system with version control for code.
SnowHill9902 4y ago
Dumb terminal + mainframe over IP?
whatever1 4y ago
Even for my home lab I use a remote setup. My local workstation/server runs ubuntu with all the bells and whistles (nginx, docker, cuda etc). My dev laptop just has vs code and connects to my local server via wifi 6 &/or gigabit ethernet. I do not notice that i develop remotely. VS code also has some great quality of life features, for example if you run a command that exposes a port in your remote machine, it automatically forwards it to your local. Same for jupyter notebooks. VSCode finally enabled the features that command line folks enjoyed for decades.
- qudat 4y ago
  I also have a remote dev machine and love it. When I do any heavy cpu work my laptop stays nice and cool since it only has a terminal and a browser open.
ilrwbwrkhv 4y ago
Uber still has time for this crap? Counting the days till they shut down. Non businesses need to die out.
Galanwe 4y ago
Can someone explain the rationale behind switching to a monorepo? I just don't get it. Does it mean also having a single unified production environment build? I have been managing a stack composed of 500+ repositories, communicating through webservices, files and ABIs for many years now, and never quite hit much of the issues cited as reasons to switch to monorepos. Having multiple small independent environments for each deployed service is a feature to me. It reduces the surface of bugs and regressions introduced by new dependencies. Not having to update dependencies globally has been a feature as well. It allows to prioritize which environments to migrate first. I found that big bang dependency updates burden is the #1 reason of _not_ updating a dependency, while allowing a per-service dependency migration ensures we can be fast to update the most important and supported services. Switching between repositories has never really been an issue to me. I found that if the structuring of projects in repositories make sense, rarely do you have to work across more than a few of them at the same time. Each repository is its own package, with its own dependencies. Features that cross repository boundaries are much less frequent than isolated ones, and updating dependencies to other projects is part of each project's PR anyway. I never tried monorepo because I never quite felt the need to. To me it seemed to be a step backward to end up with a megafat repository, where individual service history would be lost in the overall monorepo history. The deployment seems like a nightmare too, having to update the whole stack at once because you then have no idea which individual service changed between releases. Not to mention I really don't want people to spend time migrating project X - that does its work perfectly without issue since 5 years - to the latest version of LibFooBar just because project Y wants it. What am I missing?
- whydid 4y ago
  How many people are in your organization?
  - nsm 4y ago
    People seem to conflate a monorepo with having everything else the same as well, just because that's how Google and other BigCos do it. You don't have to. At some level a monorepo is just a way to stick all your code in one giant directory and manage it all under one VCS repository. You could still do separate build tools per project, separate vendoring if you really wanted and so on. However you may find that being able to simply import other first party code by path instead of doing some cross repo dependency process is a massive win. Edit: apologies, this was meant to be a reply to the top level comment.
Congeec 4y ago
I always dream about this setup. But an unreliable network prevents it. mosh and VSCode Remote partially solved the problem of latency. There's is still a problem of unavailable network - when you are in flight. nix solved my problem by and large for well modularized projects. Because, nix can provide nearly identical dev environment in practice.
- wildrhythms 4y ago
  I'm always thinking about the use case of writing code at an airport or on a plane or somewhere remote with limited (or no) reliable internet connection. Or maybe I should just take a vacation :)
siliconc0w 4y ago
I've tried to setup this up at a small tech company pre-kubernetes and there were a lot of challenges... * low utilization, so you need to auto-shutdown or auto-scale down systems but sometimes jobs did need to run overnight or on the weekend so you needed an interface/user training to avoid upsetting users/killing their jobs. * local hardware is cheap and powerful - employees are already issued really powerful laptops and some teams just went out and bought their own really, really powerful workstations. It is hard for the 'cloud' to compete with this. * bin-packing workloads was hard, Kubernetes probably solves this better. * fast-customization was hard, we had docker but it was hard to train users to update/fork the dockerfiles to keep the environment reproducible. A lot of users were more scientists than engineers and so weren't great at using version control. * persistence / shared-storage IOPs are expensive - shared storage is a nice to have and a number of teams made a lot of use of it, it also made it easy to migrate users around but it's expensive. Local disks were also very painful/slow/buggy to deattach/reattach (maybe this is better now). * latency - we needed instances close to our team and at the time the metro clusters near us were like second class with low capacity and limited features. * specialty hardware like GPUs * multiple paradigms, we also needed long running dev/staging environments, spark clusters, or other software, some managed or licensed to run in specific way, and it was hard to get these all managed the same way, clustered on the same nodes without introducing other issues - but w/out this costs would spiral. Again now that kubernetes is like the defacto cluster manager this might be easier now.
- rtpg 4y ago
  Yeah I can't really get beyond this thing about how powerful our machines on our desk are compared to what hosting providers charge if you have 12 months utilization (I get people in SV change jobs every 6 months but). I guess the thing that makes this make sense for Uber is the ginormous repo? This probably is a good "big ball of mud" solution to a bazillion tiny projects, each with varying degrees of documentation making it impossible to run. One day we as an industry will figure out how to make it so that our dev setups work and run well in a multitude of environment (hopefully without that solution involving "we are pinning to a specific Ubuntu docker image"...)
  - 4y ago
Ysraes 4y ago
I've been doing remote development over RDP for the past 3 or so years. The one problem I have is if my internet is having a bad day the latency makes using something with a UI like IntelliJ unbearable at which point I just use vim over ssh.
nitwit005 4y ago
I guess they expect Uber devs to get an extra day or two off annually from their cloud provider being down: > Unfortunately, here we are limited by the cloud provider availability that’s capped at 99.5%.
- sk0g 4y ago
  That's a big improvement over maintaining non-trivial local environments at least.
employeedeuber 4y ago
The limiting factor here is the IDE experience. You can build all this amazing architecture to provision and serve the source bits of the monorepo but if the IDE experience is janky? folks will not adopt it. The slow adoption rate speaks to that. If it truly was a productivity boom we would see a sharp spike to near 90% or more. Instead we see a slow change which is more akin to a mandate and all docs heavily suggesting for folks to use it.
- nsenifty 4y ago
  Both VS Code and IntelliJ have first class support for remote development as referenced in the article. I have been using Github codespaces for a while now and you can't nearly tell you aren't developing locally. Add to that faster builds without your laptop sounding like a jet engine.
  - employeedeuber 4y ago
    I’m not sure what repo sizes your developing in, but at big enough numbers, the set of protobuf/thrift definitions, package dependencies, and sheer number of files being looked at, brings all the remote products to their knees. IntelliSense and other syntax, highlighting, chokes. The biggest services/apps at Uber are not developed in devpods. Speculative, but these IDE’s were developed first as local first environments, there’s lots assumptions they make, adding up to terrible latency.
a-dub 4y ago
so I can't really tell from the text. are they using some remote mode in the IDEs that keeps the editor itself local or are they doing NX type stuff? also curious about the terminal access. does it maintain state on the remote (like screen/tmux) or are the sessions subject to reset under network cuts. i'd also be curious about settings since the pods are ephemeral. i suppose people would have scripts to grab their dot files, but some things like say, the android avd tool, can store settings in myriad ways. that said, looks very cool!
alganet 4y ago
Let's face it: monorepos are just huge monoliths. If you want modular software, you WILL have to pay the price of fragmentation. Someone needs to think about the boundaries between different parts of a system. If those boundaries are defined by functions, classes, packages or services, it doesn't matter that much. Yes, it is always a pain. Shipping the entire forest of modules in a single repo seems like a good compromise. It stops being a good compromise when you start creating a complicated network of soft dependencies between those modules. And when you have everything in a single repository, it's hard NOT to do that. By the way, there's nothing wrong with monoliths. They're just better if you design them as such. There's nothing wrong with microservices as well. It shouldn't be a surprise though that neither of them are actually silver bullets. I understand why these solutions were created. I see it as an ephemeral thing. Once someone figures out the tooling and practices to automate all of that painful management locally, developers will always prefer that. Then when it's fast and easy, we'll break it once again (like we did with classes, packages, containers, etc).
- ithkuil 4y ago
  Monorepos and monoliths are orthogonal Monorepos are a tooling nightmare. Monorepos are a bandwidth black hole. Monorepos make some things possible that are just not possible or very very hard when you have multiple independent repos that are built and tested independently. Namely, a) they allow you to test the effects of a change on all the components that depend on you, before you merge your change. This reduces the noise caused by regressions (API or behaviour) introduced in one component that is depended on by many consumers. b) they are a practical way to ensure that all components have up to date internal dependencies: by placing the burden of API and behaviour breakage to the author of the change, you don't end up having hundreds of teams each struggling to keep up with dependencies that keep breaking their builds when you update them and consequently hating the teams that release those changes. None of these things is a big deal unless you're a huge company with hundreds of teams. I think in theory there could be some tooling and workflow that could provide all or moat of the benefits of monorepos, without the downsides. Until then, monorepos are likely going to be a bad choice for small companies.
  - 4y ago
fizx 4y ago
This is the problem i hope WebAssembly fixes. 2GB of RAM per microservice is nuts. If we can get this back down to 10-100MB, then maybe we can run a company's dev env from a laptop again.
voz_ 4y ago
This isn’t entirely honest, as Uber doesn’t have one monorepo, it has multiple. They never understood what Mono means lol.
- lhorie 4y ago
  The joke is that we love them so much that we have many of them. In seriousness, we historically used microrepos and just getting to language-specific monorepos was already a monumental effort.
  - voz_ 4y ago
    I was on the first ever team to use or setup monorepos at Uber. We even had a banner lol. In hindsight it was the right idea but a totally wrong implementation. I wish you the best but, IMO, dividing the monorepos by language is and was a mistake.
- deleted 4y ago
  [deleted]
__MatrixMan__ 4y ago
> Controlled toolchain–pre-curated secure environment I'm a hacker dammit. I'm supposed to be dangerous if you let me within whistling distance of a payphone. The idea of slowly becoming useless without an internet connection and access to my devpod makes my skin crawl.
sammy2255 4y ago
Poor Australian devs needing to develop on Oregon with 200ms of latency
- _joel 4y ago
  It's bad enough with an ssh session to a bastion on the other side of the pond...
- ghuntley 4y ago
  Having to develop with circa 360ms pings was a contributing factor to me deciding to leave Gitpod. https://ghuntley.com/tea https://ghuntley.com/tea
  - Dave3of5 4y ago
    Wow that's insane glad I decided not to use gitpod.
    - debosmit 4y ago
      fwiw, we’re solving this at DevZero with having multiple AZs - given that our platform centrally manages remote compute (hibernation etc), this is working great at making sure latencies are <20ms for users
lukewiwa 4y ago
I've definitely used the vscode remote ssh functionality before when I've had to use a particular architecture (x86) while I was developing on an M1 mac. In cases like this I already have the dev environment configuration setup for local dev so it's super easy to spin up an EC2 instance and I'm off to the races. I definitely see the use case, but in saying that I find local development really valuable and default to it when I can. I do however run dev work almost exclusively inside a container so I'm flexible either way. I can see how some might not be.
tedd4u 4y ago
There's at least one good commercial solution that basically does the same thing the blog describes [1]. Working well for us so far. [1] https://www.gitpod.io https://www.gitpod.io
debosmit 4y ago
[Putting a note here for awareness. I am debo, a user/contributor to the devpods project at Uber] I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback! How we're looking at the space: - IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers - the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against. The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here. For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc). When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode". Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev. Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring). Re: workload management, we added support for k8s* generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc). *Say you have helm to deploy containers/pods to prod. We look at that (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.
debosmit 4y ago
[Putting a note here for awareness. I am debo, a user/contributor to the devpods project at Uber] I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback! How we're looking at the space: - IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers - the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against. The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here. For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management*" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc). When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode". Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev. Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring). *Re: workload management, we added support for k8s** generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc). **Say you have helm to deploy containers/pods to prod. We take those charts (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.
sandGorgon 4y ago
genuine question here. im hoping u folks have put a lot more thought into it than i have. so devpod "production OS" - the top level devpod - contains ALL running services ? at Uber scale it is what 64 GB RAM ? if ur already using kubernetes ... why do it this way rather than have kubernetes namespaces with many containers in a dev cluster ? trivially this is a docker compose stack right ? second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more.
- fragmede 4y ago
  > at Uber scale it is what 64 GB RAM ? I don't work there, but I bet their full stack takes more than 64 GiB RAM. > trivially this is a docker compose stack right ? In the way that docker is the same thing as Kubernetes, yes. However there are differences that become material when you zoom in a bit closer. second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more. > second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more. I don't think it's an optimization for dev laptop specs. At the end of the day, it's cheap for Uber to just max out the ram on dev laptops.
sierkov 4y ago
> The Linux file system, which performs better compared to laptop file systems ... say what?
- yjftsjthsd-h 4y ago
  Probably means "our laptops only run Windows, which infamously has performance issues with git because of different filesystem semantics (something about caching metadata?), but Linux filesystem drivers don't have that problem".
  - ccouzens 4y ago
    Or they were using Docker on Mac
nathants 4y ago
using remote resources as a part of your local dev flow can be very useful if your local environment is constrained on cpu/ram/gpu/ssd/bandwidth. this can be as simple as an ephemeral ec2 spot machine that reacts every time files on it’s filesystem change. it then does stuff, like building and shipping. your local setup needs to rsync files from local to remote every time you save a file. i’m on an upload constrained setup right now, and this[1] significantly speeds up my iterations uploading lambda zips. fancier setups probably are similarly advantageous, but add tradeoffs proportional to their complexity. 1. https://github.com/nathants/aws-gocljs/blob/258ea5bb72d06a5041174b7ef4522b40e21eaf8e/bin/relay.sh https://github.com/nathants/aws-gocljs/blob/258ea5bb72d06a50...
decodebytes 4y ago
This just seems like a lot of cloud costs to take on, considering how much power developers laptops carry and would be completely untapped. Instead of having everything remote, to me it appears more sensible to have a distributed development environment. I guess this would look something like the dagger.io folks are shooting for.
- yjftsjthsd-h 4y ago
  I think the steelman answer is that if this works well, you don't give developers powerful laptops; you give them cheap laptops that are little more than dumb terminals. (Whether that works is left as an exercise for the reader.)
- debosmit 4y ago
  we’re not seeing cloud costs be too terrible at DevZero - with proper hibernation/suspension, cloud costs are ~$50-60/mo in the worst case admittedly, we target only enterprise companies where the cost of loss of dev efficiencies is much higher Say net cost to company for an engineer is $100k-$200k+. Even a net 10% savings over a year means $10k-$20k+ vs a $600-1k/yr investment (in worst case). Security posture is also significantly improved, which admittedly is harder to assign a $ value to
dvh 4y ago
I was expecting towable spherical thing with solar panels, eco toilet and a hammock
ghuntley 4y ago
I’m one of the engineers at Coder and we build internal development platforms for a living. Here’s how our team builds Coder with Coder - https://coder.com/blog/how-our-development-team-shares-one-giant-bare-metal-machine https://coder.com/blog/how-our-development-team-shares-one-g... Over at https://GitHub.com/coder/coder https://GitHub.com/coder/coder you’ll find our source code btw. Essentially we provision software development environments using terraform for Linux,windows,Mac,arm,amd64 and soon FreeBSD…
xiwenc 4y ago
At my current startup we are doing something similar. Perhaps as inspiration for others without the huge budgets. From very high level it works as follow: - developer logs in to AWS cli - executes: dev/01-start-env.sh - prepare infra services: dev/02-base-platform.sh - code can be changed and run locally. But if they need to test in the bigger system: dev/03-deploy-code.sh There is dev/99-delete-env.sh An environment is a personal ec2 spot instance that auto shutsdown if there’s no developer activity for over an hour. The idea is that all developers work (program/code) locally as much as possible. But deploys changes to their own private environment in a much heavier VM’s that runs all services in containers. Also there is a dev/tests.sh that executes the exact same test cases as in continuous integration. In fact, we try to bring all checks enforced during CI to be available to developers in their semi-remote-dev area environments for quick feedbacks.
deleted 4y ago
[deleted]
tomjen3 4y ago
This seems like a massive amount of work and frustration instead of admitting that mono repo doesn't work and scrapping that. Plus super frustrating to have somebody preconfigure your environment because they will inevitably get it wrong.
poisonta 4y ago
So, they have services than engineers - 4000+ services and 3000 engineers? So, some services don't have maintainers?
exelib 4y ago
Off-topic question: What are 70 Mio. code lines for? I mean, yeah, there are some services like Uber, Uber eats, but gosh, 70 Mio.? I have worked on very sophisticated products with a bit more than 1 Mio. LOC... Do I miss something in Uber apps and services?
- Beltalowda 4y ago
  Or "4000+ services", and "500+ Web Apps", and the many thousands of devs they employ. I've wondered for quite a while where all this code and complexity is at. My guess is a lot of it is in their self-driving cars projects and stuff like this, but I don't really know.
AJRF 4y ago
Anyone from Uber here that works on this? You ever expect a DevPod for iOS developers? I've been toying with idea of doing this in work for our devs on Intel MacBooks, using -> https://github.com/sickcodes/Docker-OSX https://github.com/sickcodes/Docker-OSX, which enables local running on iOS devices. I think it's hampered by licensing though, maybe that's why it's not used or mentioned in this post.
- chris-laffra 4y ago
  I worked on this at Uber in the past and you explain very well why there is no iOS flavor for devpods.
- debosmit 4y ago
  We’ve thought about this quite deeply at DevZero. Yes, the iOS developer experience can be quite hampered on the local env and requires users to frequently get beefy local laptops/machines. I saw this first hand at Uber from when crowdstrike was rolled out broadly. We have support for AWS-based Mac VMs on DevZero but we don’t find our customers having their biggest issues related to iOS dev yet (we also target enterprise cos that have a vast diversity of tools, mostly backend and front end)
franole 4y ago
I was getting a 404 error. Apparently, because the site redirects me to a non-existent localized version of the page. You can bypass this by forcing English in the URL https://www.uber.com/en-US/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/en-US/blog/devpod-improving-developer-p...
juancn 4y ago
These things usually suck when latency is high... even if you are supporting most major regions it's not usual to find yourself in a situation when you have a few hundred ms latency. Anything over ~40ms is very noticeable when typing and editing. Then you're screwed.
SergeAx 4y ago
And this, ladies and gentlemen, is where monorepos will inevitably take you. There's actually an elementary litmus test on migrating to monorepos. It is a questionnaire with only one question: is your company Google? If the answer is "no" — you don't need a monorepo. You are welcome.

291 points by phgn 4y ago

TLDR for vscode remote users? Let me be lazy and just get a Y/N over whether it has any benefit over the normal remote vscode UX.

lozenge 4y ago
They reimplemented GitHub Codespaces without GitHub

trynewideas 4y ago

Very curious how, or if, this helps or affects non-developers also working in the monorepo, like tech writers, QA, UX/UI/visual designers, training/enablement, support, etc.

cloudking 4y ago

This sounds awesome, but not practical for smaller companies. The cloud costs must be significant? "Faster Git, Build, and IDE experience by utilizing beefy cloud resources–up to 48 cores and 96 GB of RAM per environment"

jeffbee 4y ago
Why would it need to be per user? An environment of that size could easily support many users. Developers spend most of their time just staring and only intermittently need to build. There is no particular reason why everyone would be building at once. Most of the time you'll get all those cores to yourself even though there could be dozens of devs sharing the box.
- gabereiser 4y ago
  We’re coming around full circle to the old days of thin client terminals to a beefy mainframe under the corporation’s control. Only it’s hosted in the cloud and all your data is at risk.
  - dilyevsky 4y ago
    You can run it not in cloud. Uber uses its own DCs (although not sure they do that for dev). It’ll probably be more expensive because you can’t scale everything down for the night (unless you have dev teams spread globally).
- epolanski 4y ago
  Also, smaller company can probably go a long way just on codespaces. If GitHub can, 99% of projects out there likely can too.
cyc116 4y ago
Does smaller companies have mono repos that size?
Moto7451 4y ago
Correct. My last job built a low rent version of this and it was very painful. 15-45 minutes to launch a pod that would have taken 3 minutes under Compose. Pushed a new base image? Whole thing would roll over and need another 15-45 minutes. To do this well it takes manpower. There are some off the shelf options out there for remote/prod like Kubernetes based development that are smarter choices than building your own. The table stakes for doing this is being a big corp that can afford to spend millions on one custom tool with a hope for positive ROI. For the rest of us, find something that works that is open source or easily licensed.

deleted 4y ago

[deleted]

ValleZ 4y ago

Most developers in Uber use local builds anyway because of input lag. Dev pods were mostly good to prevent laptop from working at 100% fan volume at all times though.

mrhands556 4y ago
input lag like when typing? I use remote env on vscode and afaik it’s locally editing and syncing after the fact for lint, compilation etc
- judge2020 4y ago
  It was probably just a RDP connection, which either signals how old this insight is, or how far behind Uber is on these new ways to remotely code.
  - esprehn 4y ago
    The post clearly shows they're not using RDP. That said Projector round tripped all the key strokes to get new draw commands (unlike VSCode) which also results in lag. The new IntelliJ remote architecture is much better, and it seems Uber is moving that way too.
    - judge2020 4y ago
      They're using this now, but I imagine the user ValleZ's comment is about a past time working at Uber or hearing stories from Uber developer friends, in which I can see why spinning up a low-latency (but still over 30+ millisecond RTT) virtual machine would be an early solution to developers trying to cut down on build times and whatnot.
    - Noumenon72 4y ago
      What's the new way? JetBrains Gateway? Have they caught up to VSCode's devcontainers?
WookieRushing 4y ago

prakashn27 4y ago

We already have something like this at Meta for improving developer experience. All our dev environments are remote

fragmede 4y ago
Any externally visible docs on it? Google's is documented at https://cloud.google.com/blog/topics/developers-practitioners/googles-virtual-desktop-future https://cloud.google.com/blog/topics/developers-practitioner...
- nsenifty 4y ago
  I don't think it's been discussed externally yet. It's called on-demand (some references to it here - https://developers.facebook.com/blog/post/2022/11/15/meta-developers-workflow-exploring-tools-used-to-code/ https://developers.facebook.com/blog/post/2022/11/15/meta-de...).
  - fragmede 4y ago
    Fascinating, thanks!

xnx 4y ago

Similar setup at Google: https://cloud.google.com/blog/topics/developers-practitioners/googles-virtual-desktop-future https://cloud.google.com/blog/topics/developers-practitioner...

jeffbee 4y ago
The contrast between the Uber and Google setups would be twofold. Google engineers weren't doing local builds anyway (unless they intentionally inflicted those upon themselves) because of Forge. The Forge builders already had 10000 CPU cores in 2010[1]. You can probably imagine how many cores Forge or its equivalent has today. Secondly Google doesn't suffer from the performance problems of a weird freeware VCS, because they don't use git or the git workflow model at all. They built their own VCS based on the workflow model from Perforce. One funny aspect of Google moving development into cloud machines was that a decade before the process ran the other way. The desktop that was issued to most engineers was a production websearch machine turned sideways and stuck under your desk. 1: https://youtu.be/b52aXZ2yi08?t=1118 https://youtu.be/b52aXZ2yi08?t=1118
- ridiculous_fish 4y ago
  Google contains multitudes. Android, Chrome, and ChromeOS are examples of large Google projects that do not use Perforce and are routinely built locally.
  - hoten 4y ago
    Local-ish. (at least) Chromium is built by Googlers w/ the aide of goma, basically a distcc[1]. [1] https://chromium.googlesource.com/chromium/src/+/778a7e84f65fd36658f91881bda07b9153a8729d/docs/linux_faster_builds.md#use-goma https://chromium.googlesource.com/chromium/src/+/778a7e84f65...
  - ameliaquining 4y ago
    Chrome, at least, is typically built in the cloud (I don't know about Android, though it would be weird for it to be the only major project that doesn't do this). It's not just that you can have beefier machines in the cloud (though that's a factor); it's also that you can have a shared cache of build artifacts for all your users, which makes most workflows a lot faster. See documentation: https://chromium.googlesource.com/infra/goma/client/ https://chromium.googlesource.com/infra/goma/client/

deleted 4y ago

[deleted]

Gigachad 4y ago

You will develop in the pod. You will not own a capable laptop.

headbansown 4y ago
You will fix the bugs?

teeray 4y ago

This sounds great. Certainly the IP zealots love it because the code never leaves the walled garden, and it’s a super-secure blah blah environment. Consistent tooling, easy onboarding… you get the idea. All good until you have an outage. Then your development team’s productivity drops to exactly zero while you fix it and your entire production environment is now potentially vulnerable to defects you can’t fix until the development environment is fixed (better hope it stays running). This is when companies realize that the development environment is actually a service and it needs higher SLA targets than production, but it will never get the attention that it needs to achieve those targets (because it’s just dev, right?).

gandalfgeek 4y ago
Every service has the risk of introducing a new point of failure, of course. But compare dev productivity lost due to service downtime with that of each new SWE in your org burning time to A) setup their own unique snowflake of an environment and B) futzing and debugging it when it breaks or there's a software update.
- erik_seaberg 4y ago
  > burning time to setup their own unique snowflake of an environment Anyone good is doing that even if you tell him not to. One size does not fit all.
charcircuit 4y ago
You already have the same concern with source control, code review, continuous integration, continuous deployment, etc. Dev servers are arguably much simpler to provide without issues.
- teeray 4y ago
  But I don’t really have those concerns—they’re better mitigated. If source control goes down, I have a complete clone on my machine and so do my coworkers (yay git). I have done code reviews by pasting `git am` formatted patches between coworkers during particularly long GitHub outages. CI/CD is more problematic, but I only need those things to integrate work. I can still do work and verify correctness with tests that I can run locally. None of that is possible in a centrally-managed dev VM setup. When the VMs go down, you send your dev team home until it’s fixed. You’re still paying their salaries while you pay another team to make them productive again.

gandalfgeek 4y ago

Back in the early 2000s, Sun ("the network is the computer") had a similar solution that worked seamlessly for most of their software org-- the Sun Ray. https://en.wikipedia.org/wiki/Sun_Ray https://en.wikipedia.org/wiki/Sun_Ray It was a network terminal. Your files and entire session were on the server. Your “local” terminal consisted only of a network interface and enough compute power to display your session. The way they had it set up was that you could insert your Sun employee ID – the same card used to get into the building – into a slot in the terminal. That authenticated you to the server and displayed your session instantly. Want to show a colleague something you’re working on? Just put your ID into their Sun Ray and show them exactly what you were doing. That was cool! It was a frictionless way to demo and collaborate.

jimmaswell 4y ago
What's old is new again. How soon until we realize the X window system actually had some good ideas again and start running desktop apps on cloud servers for remote work?
- erik_seaberg 4y ago
  X maintainers are promoting a new platform that doesn’t provide for remote hardware rendering. Ironically, shipping Javascript to a web browser (local to the display) seems to be the way ahead for performance.
- paxys 4y ago
  It had some good ideas, but that was the extent of it. Every iteration of the solution since then has agreed that streaming all application UI is always going to be janky and needless. It's a lot better to host the data on the server side and let the client handle visual rendering by itself.
  - jimmaswell 4y ago
    I used to run Firefox on my Debian server with X tunneling in high school sometimes. It wasn't that bad then and surely it's even better with today's internet.

EastSmith 4y ago

We are moving to such cloud environment, and it makes me sick. Maybe you need to dockerise Mongo, MySql and 5 other dependencies - I can get this, but I don't get it why the rest the code should still be running in the cloud. Python, Rails, Node? Why? Developers should be able to run 1 shell commands to install node. Dev experience excuses are just excuses for a bad setup. So, fix your setup, please. Not being able to run the project natively is big red flag for me and when I move jobs will be my 1st question.

why5s 4y ago
For work-related stuff: I disagree. We're not even at Uber's scale at my company and I hate having to manage the ever-changing set of dependencies that I have no control over.
- layer8 4y ago
  The thing is, it doesn’t have to be ever-changing, and creating a reproducible working setup locally can be achieved by appropriate setup scripts.
  - paxys 4y ago
    Easy in theory, but always breaks down for a non-trivial project with >5 developers on it.
  - tylerhou 4y ago
    Setup scripts that mutate an operating system are fragile. Break something? Unless you understand how the scripts work (which will become stupidly complex at a scale like Uber’s) you will have to reinstall your machine. Or you will have to staff a ton of support to help users when they do break their configuration. Spinning up a VM with an image containing all the development tools is a much smoother experience most of the time. The only reason why I don’t use it where I work is because I use vim and network adds too much latency for me.
    - 4y ago

faizshah 4y ago

I like this workflow, use something like mutagen to sync from your local environment to the remote machine. For me, I prefer jetbrains IDEs so I tried using jetbrains gateway, it’s really buggy and slow. I prefer to use jetbrains ides on my local machine and use VSCode remote for interacting with the remote machine. I wonder though, is it worth it to setup a remote dev machine if you have an M1 Mac? I think you can get good compile times for Rust etc. on an M1. For me I have an intel Mac at work and at home so the remote dev env is better for builds.

hokumguru 4y ago
I think there are still some limits applied even on cutting edge hardware. GitHub previously said they hit network limitations cloning their multi-gb repo for instance.
- mejutoco 4y ago
  If I am not mistaken, git clone is an atomic operation (it either succeeds or fails). Since I see it mentioned in many threads here: for a huge repo one can always use `git clone --depth 1` to get that repo and later do a proper pull to retrieve all history.
nithayakumar 4y ago
Yeah - we've (usenimbus.com - a company in this space) have heard the same about Jetbrains IDEs. You should check out our extension (or others') because there's been a lot of work put into making performance better. M1s are a mixed bag for this way of working. Pre-M1, devs were running into local computing power issues. Post-M1, more compatibility and stability issues.

quechimba 4y ago

For some reason this link redirects to https://www.uber.com/es-CO/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/es-CO/blog/devpod-improving-developer-p... which results in a 404. This URL works without redirects: https://www.uber.com/en-US/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/en-US/blog/devpod-improving-developer-p...

emptysea 4y ago

Wonder how it handles things like dot files? Also does it automatically install dependencies? Feel like running npm install each time (or the equivalent) would be slow.

chazeon 4y ago

Glad to see industry adopting cloud server + ssh + code-server / code-remote as a standard. As a PhD student I have been working on HPCs like this with an environment I built myself for a while. It is really good and helps you to move fast.

icedchai 4y ago
It depends on latency. If you have a slow network connection (or are simply far away from the cloud data center) you are in for pain. That being said, running a local environment on a constantly throttling 2019 MacBook Pro that sounds like a jet engine is probably worse.

eddsh1994 4y ago

This sounds very similar to my first IT Support role setting up thin clients for law firms in London that had VM instances for each role including development, and then a networked file system with version control for code.

SnowHill9902 4y ago

Dumb terminal + mainframe over IP?

whatever1 4y ago

Even for my home lab I use a remote setup. My local workstation/server runs ubuntu with all the bells and whistles (nginx, docker, cuda etc). My dev laptop just has vs code and connects to my local server via wifi 6 &/or gigabit ethernet. I do not notice that i develop remotely. VS code also has some great quality of life features, for example if you run a command that exposes a port in your remote machine, it automatically forwards it to your local. Same for jupyter notebooks. VSCode finally enabled the features that command line folks enjoyed for decades.

qudat 4y ago
I also have a remote dev machine and love it. When I do any heavy cpu work my laptop stays nice and cool since it only has a terminal and a browser open.

ilrwbwrkhv 4y ago

Uber still has time for this crap? Counting the days till they shut down. Non businesses need to die out.

Galanwe 4y ago

Can someone explain the rationale behind switching to a monorepo? I just don't get it. Does it mean also having a single unified production environment build? I have been managing a stack composed of 500+ repositories, communicating through webservices, files and ABIs for many years now, and never quite hit much of the issues cited as reasons to switch to monorepos. Having multiple small independent environments for each deployed service is a feature to me. It reduces the surface of bugs and regressions introduced by new dependencies. Not having to update dependencies globally has been a feature as well. It allows to prioritize which environments to migrate first. I found that big bang dependency updates burden is the #1 reason of _not_ updating a dependency, while allowing a per-service dependency migration ensures we can be fast to update the most important and supported services. Switching between repositories has never really been an issue to me. I found that if the structuring of projects in repositories make sense, rarely do you have to work across more than a few of them at the same time. Each repository is its own package, with its own dependencies. Features that cross repository boundaries are much less frequent than isolated ones, and updating dependencies to other projects is part of each project's PR anyway. I never tried monorepo because I never quite felt the need to. To me it seemed to be a step backward to end up with a megafat repository, where individual service history would be lost in the overall monorepo history. The deployment seems like a nightmare too, having to update the whole stack at once because you then have no idea which individual service changed between releases. Not to mention I really don't want people to spend time migrating project X - that does its work perfectly without issue since 5 years - to the latest version of LibFooBar just because project Y wants it. What am I missing?

whydid 4y ago
How many people are in your organization?
- nsm 4y ago
  People seem to conflate a monorepo with having everything else the same as well, just because that's how Google and other BigCos do it. You don't have to. At some level a monorepo is just a way to stick all your code in one giant directory and manage it all under one VCS repository. You could still do separate build tools per project, separate vendoring if you really wanted and so on. However you may find that being able to simply import other first party code by path instead of doing some cross repo dependency process is a massive win. Edit: apologies, this was meant to be a reply to the top level comment.

Congeec 4y ago

I always dream about this setup. But an unreliable network prevents it. mosh and VSCode Remote partially solved the problem of latency. There's is still a problem of unavailable network - when you are in flight. nix solved my problem by and large for well modularized projects. Because, nix can provide nearly identical dev environment in practice.

wildrhythms 4y ago
I'm always thinking about the use case of writing code at an airport or on a plane or somewhere remote with limited (or no) reliable internet connection. Or maybe I should just take a vacation :)

siliconc0w 4y ago

I've tried to setup this up at a small tech company pre-kubernetes and there were a lot of challenges... * low utilization, so you need to auto-shutdown or auto-scale down systems but sometimes jobs did need to run overnight or on the weekend so you needed an interface/user training to avoid upsetting users/killing their jobs. * local hardware is cheap and powerful - employees are already issued really powerful laptops and some teams just went out and bought their own really, really powerful workstations. It is hard for the 'cloud' to compete with this. * bin-packing workloads was hard, Kubernetes probably solves this better. * fast-customization was hard, we had docker but it was hard to train users to update/fork the dockerfiles to keep the environment reproducible. A lot of users were more scientists than engineers and so weren't great at using version control. * persistence / shared-storage IOPs are expensive - shared storage is a nice to have and a number of teams made a lot of use of it, it also made it easy to migrate users around but it's expensive. Local disks were also very painful/slow/buggy to deattach/reattach (maybe this is better now). * latency - we needed instances close to our team and at the time the metro clusters near us were like second class with low capacity and limited features. * specialty hardware like GPUs * multiple paradigms, we also needed long running dev/staging environments, spark clusters, or other software, some managed or licensed to run in specific way, and it was hard to get these all managed the same way, clustered on the same nodes without introducing other issues - but w/out this costs would spiral. Again now that kubernetes is like the defacto cluster manager this might be easier now.

rtpg 4y ago
Yeah I can't really get beyond this thing about how powerful our machines on our desk are compared to what hosting providers charge if you have 12 months utilization (I get people in SV change jobs every 6 months but). I guess the thing that makes this make sense for Uber is the ginormous repo? This probably is a good "big ball of mud" solution to a bazillion tiny projects, each with varying degrees of documentation making it impossible to run. One day we as an industry will figure out how to make it so that our dev setups work and run well in a multitude of environment (hopefully without that solution involving "we are pinning to a specific Ubuntu docker image"...)
- 4y ago

Ysraes 4y ago

I've been doing remote development over RDP for the past 3 or so years. The one problem I have is if my internet is having a bad day the latency makes using something with a UI like IntelliJ unbearable at which point I just use vim over ssh.

nitwit005 4y ago

I guess they expect Uber devs to get an extra day or two off annually from their cloud provider being down: > Unfortunately, here we are limited by the cloud provider availability that’s capped at 99.5%.

sk0g 4y ago
That's a big improvement over maintaining non-trivial local environments at least.

employeedeuber 4y ago

The limiting factor here is the IDE experience. You can build all this amazing architecture to provision and serve the source bits of the monorepo but if the IDE experience is janky? folks will not adopt it. The slow adoption rate speaks to that. If it truly was a productivity boom we would see a sharp spike to near 90% or more. Instead we see a slow change which is more akin to a mandate and all docs heavily suggesting for folks to use it.

nsenifty 4y ago
Both VS Code and IntelliJ have first class support for remote development as referenced in the article. I have been using Github codespaces for a while now and you can't nearly tell you aren't developing locally. Add to that faster builds without your laptop sounding like a jet engine.
- employeedeuber 4y ago
  I’m not sure what repo sizes your developing in, but at big enough numbers, the set of protobuf/thrift definitions, package dependencies, and sheer number of files being looked at, brings all the remote products to their knees. IntelliSense and other syntax, highlighting, chokes. The biggest services/apps at Uber are not developed in devpods. Speculative, but these IDE’s were developed first as local first environments, there’s lots assumptions they make, adding up to terrible latency.

a-dub 4y ago

so I can't really tell from the text. are they using some remote mode in the IDEs that keeps the editor itself local or are they doing NX type stuff? also curious about the terminal access. does it maintain state on the remote (like screen/tmux) or are the sessions subject to reset under network cuts. i'd also be curious about settings since the pods are ephemeral. i suppose people would have scripts to grab their dot files, but some things like say, the android avd tool, can store settings in myriad ways. that said, looks very cool!

alganet 4y ago

Let's face it: monorepos are just huge monoliths. If you want modular software, you WILL have to pay the price of fragmentation. Someone needs to think about the boundaries between different parts of a system. If those boundaries are defined by functions, classes, packages or services, it doesn't matter that much. Yes, it is always a pain. Shipping the entire forest of modules in a single repo seems like a good compromise. It stops being a good compromise when you start creating a complicated network of soft dependencies between those modules. And when you have everything in a single repository, it's hard NOT to do that. By the way, there's nothing wrong with monoliths. They're just better if you design them as such. There's nothing wrong with microservices as well. It shouldn't be a surprise though that neither of them are actually silver bullets. I understand why these solutions were created. I see it as an ephemeral thing. Once someone figures out the tooling and practices to automate all of that painful management locally, developers will always prefer that. Then when it's fast and easy, we'll break it once again (like we did with classes, packages, containers, etc).

ithkuil 4y ago
Monorepos and monoliths are orthogonal Monorepos are a tooling nightmare. Monorepos are a bandwidth black hole. Monorepos make some things possible that are just not possible or very very hard when you have multiple independent repos that are built and tested independently. Namely, a) they allow you to test the effects of a change on all the components that depend on you, before you merge your change. This reduces the noise caused by regressions (API or behaviour) introduced in one component that is depended on by many consumers. b) they are a practical way to ensure that all components have up to date internal dependencies: by placing the burden of API and behaviour breakage to the author of the change, you don't end up having hundreds of teams each struggling to keep up with dependencies that keep breaking their builds when you update them and consequently hating the teams that release those changes. None of these things is a big deal unless you're a huge company with hundreds of teams. I think in theory there could be some tooling and workflow that could provide all or moat of the benefits of monorepos, without the downsides. Until then, monorepos are likely going to be a bad choice for small companies.
- 4y ago

fizx 4y ago

This is the problem i hope WebAssembly fixes. 2GB of RAM per microservice is nuts. If we can get this back down to 10-100MB, then maybe we can run a company's dev env from a laptop again.

voz_ 4y ago

This isn’t entirely honest, as Uber doesn’t have one monorepo, it has multiple. They never understood what Mono means lol.

lhorie 4y ago
The joke is that we love them so much that we have many of them. In seriousness, we historically used microrepos and just getting to language-specific monorepos was already a monumental effort.
- voz_ 4y ago
  I was on the first ever team to use or setup monorepos at Uber. We even had a banner lol. In hindsight it was the right idea but a totally wrong implementation. I wish you the best but, IMO, dividing the monorepos by language is and was a mistake.
deleted 4y ago
[deleted]

__MatrixMan__ 4y ago

> Controlled toolchain–pre-curated secure environment I'm a hacker dammit. I'm supposed to be dangerous if you let me within whistling distance of a payphone. The idea of slowly becoming useless without an internet connection and access to my devpod makes my skin crawl.

sammy2255 4y ago

Poor Australian devs needing to develop on Oregon with 200ms of latency

_joel 4y ago
It's bad enough with an ssh session to a bastion on the other side of the pond...
ghuntley 4y ago
Having to develop with circa 360ms pings was a contributing factor to me deciding to leave Gitpod. https://ghuntley.com/tea https://ghuntley.com/tea
- Dave3of5 4y ago
  Wow that's insane glad I decided not to use gitpod.
  - debosmit 4y ago
    fwiw, we’re solving this at DevZero with having multiple AZs - given that our platform centrally manages remote compute (hibernation etc), this is working great at making sure latencies are <20ms for users

lukewiwa 4y ago

I've definitely used the vscode remote ssh functionality before when I've had to use a particular architecture (x86) while I was developing on an M1 mac. In cases like this I already have the dev environment configuration setup for local dev so it's super easy to spin up an EC2 instance and I'm off to the races. I definitely see the use case, but in saying that I find local development really valuable and default to it when I can. I do however run dev work almost exclusively inside a container so I'm flexible either way. I can see how some might not be.

tedd4u 4y ago

There's at least one good commercial solution that basically does the same thing the blog describes [1]. Working well for us so far. [1] https://www.gitpod.io https://www.gitpod.io

debosmit 4y ago

[Putting a note here for awareness. I am debo, a user/contributor to the devpods project at Uber] I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback! How we're looking at the space: - IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers - the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against. The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here. For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc). When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode". Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev. Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring). Re: workload management, we added support for k8s* generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc). *Say you have helm to deploy containers/pods to prod. We look at that (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.

debosmit 4y ago

[Putting a note here for awareness. I am debo, a user/contributor to the devpods project at Uber] I am a founder of DevZero (devzero.io) where we are taking the theme of "remote compute with local tools", but built specifically to serve engineers in enterprise companies - still pretty early, but would love for people to check it out and provide feedback! How we're looking at the space: - IDEs need to stay local but thankfully, VS Code, Jetbrains etc all now allow connecting to remote VMs and containers - the main issue is around not have enough of your dependencies present. So outside of standalone dev environment (VM/container), we also let companies "bring their k8s/serverless config" and let each engineer have their ephemeral full-stack to code against. The standalone environments offer various perf boosts (super simple onboarding, switching project, we're seeing it reduces net time-to-deploy from start to deployed as well) for engineers coding in monoliths, which is true in many large companies still. We're already seeing really good traction here. For the stuff where we're trying to take the engineer's IDE to an "ephemeral and hermetic env" that is built off of however "production workload management*" works: an engineer can connect their local IDE to a remote "devpod" and do all their normal coding activities (w/ the relevant boosts in speed etc). When they want to do some form of end-to-end testing, the engineer can hit downstream pods/serverless stacks etc (their own copy, i.e., not shared tenancy). We're currently figuring out if we can enable engineers to test a full end-to-end call chain and connect live debuggers to arbitrary pods in that call chain -- I think this will make debugging amazing cause so far its been pretty hard to repro end-to-end call chains in "dev-mode". Our platform approach is basically making cloud dev environments (CDEs) even more awesome by putting them within a production-like environments for every dev. Lots of info here (not shared widely yet, not even on the website). Please let us know if you want to use it, or consider working with us (we're actively hiring). *Re: workload management, we added support for k8s** generally but are now expanding out across the various clouds. For serverless, we started with AWS lambda and now have to tackle the other clouds. Then, we also need to do the default container mgmt for each of the cloud providers. (also looking at hashicorp nomad etc). **Say you have helm to deploy containers/pods to prod. We take those charts (and w/ a little more config re: dbs etc), give every engineer their copy of prod in their namespace alongside a devpod. Similar vibes for other workload mgmt systems.

sandGorgon 4y ago

genuine question here. im hoping u folks have put a lot more thought into it than i have. so devpod "production OS" - the top level devpod - contains ALL running services ? at Uber scale it is what 64 GB RAM ? if ur already using kubernetes ... why do it this way rather than have kubernetes namespaces with many containers in a dev cluster ? trivially this is a docker compose stack right ? second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more.

fragmede 4y ago
> at Uber scale it is what 64 GB RAM ? I don't work there, but I bet their full stack takes more than 64 GiB RAM. > trivially this is a docker compose stack right ? In the way that docker is the same thing as Kubernetes, yes. However there are differences that become material when you zoom in a bit closer. second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more. > second question - has there been a ROI recovery in terms of laptop hardware for devs ? like - u only 8 GB ram laptops and not more. I don't think it's an optimization for dev laptop specs. At the end of the day, it's cheap for Uber to just max out the ram on dev laptops.

sierkov 4y ago

> The Linux file system, which performs better compared to laptop file systems ... say what?

yjftsjthsd-h 4y ago
Probably means "our laptops only run Windows, which infamously has performance issues with git because of different filesystem semantics (something about caching metadata?), but Linux filesystem drivers don't have that problem".
- ccouzens 4y ago
  Or they were using Docker on Mac

nathants 4y ago

using remote resources as a part of your local dev flow can be very useful if your local environment is constrained on cpu/ram/gpu/ssd/bandwidth. this can be as simple as an ephemeral ec2 spot machine that reacts every time files on it’s filesystem change. it then does stuff, like building and shipping. your local setup needs to rsync files from local to remote every time you save a file. i’m on an upload constrained setup right now, and this[1] significantly speeds up my iterations uploading lambda zips. fancier setups probably are similarly advantageous, but add tradeoffs proportional to their complexity. 1. https://github.com/nathants/aws-gocljs/blob/258ea5bb72d06a5041174b7ef4522b40e21eaf8e/bin/relay.sh https://github.com/nathants/aws-gocljs/blob/258ea5bb72d06a50...

decodebytes 4y ago

This just seems like a lot of cloud costs to take on, considering how much power developers laptops carry and would be completely untapped. Instead of having everything remote, to me it appears more sensible to have a distributed development environment. I guess this would look something like the dagger.io folks are shooting for.

yjftsjthsd-h 4y ago
I think the steelman answer is that if this works well, you don't give developers powerful laptops; you give them cheap laptops that are little more than dumb terminals. (Whether that works is left as an exercise for the reader.)
debosmit 4y ago
we’re not seeing cloud costs be too terrible at DevZero - with proper hibernation/suspension, cloud costs are ~$50-60/mo in the worst case admittedly, we target only enterprise companies where the cost of loss of dev efficiencies is much higher Say net cost to company for an engineer is $100k-$200k+. Even a net 10% savings over a year means $10k-$20k+ vs a $600-1k/yr investment (in worst case). Security posture is also significantly improved, which admittedly is harder to assign a $ value to

dvh 4y ago

I was expecting towable spherical thing with solar panels, eco toilet and a hammock

ghuntley 4y ago

I’m one of the engineers at Coder and we build internal development platforms for a living. Here’s how our team builds Coder with Coder - https://coder.com/blog/how-our-development-team-shares-one-giant-bare-metal-machine https://coder.com/blog/how-our-development-team-shares-one-g... Over at https://GitHub.com/coder/coder https://GitHub.com/coder/coder you’ll find our source code btw. Essentially we provision software development environments using terraform for Linux,windows,Mac,arm,amd64 and soon FreeBSD…

xiwenc 4y ago

At my current startup we are doing something similar. Perhaps as inspiration for others without the huge budgets. From very high level it works as follow: - developer logs in to AWS cli - executes: dev/01-start-env.sh - prepare infra services: dev/02-base-platform.sh - code can be changed and run locally. But if they need to test in the bigger system: dev/03-deploy-code.sh There is dev/99-delete-env.sh An environment is a personal ec2 spot instance that auto shutsdown if there’s no developer activity for over an hour. The idea is that all developers work (program/code) locally as much as possible. But deploys changes to their own private environment in a much heavier VM’s that runs all services in containers. Also there is a dev/tests.sh that executes the exact same test cases as in continuous integration. In fact, we try to bring all checks enforced during CI to be available to developers in their semi-remote-dev area environments for quick feedbacks.

deleted 4y ago

[deleted]

tomjen3 4y ago

This seems like a massive amount of work and frustration instead of admitting that mono repo doesn't work and scrapping that. Plus super frustrating to have somebody preconfigure your environment because they will inevitably get it wrong.

poisonta 4y ago

So, they have services than engineers - 4000+ services and 3000 engineers? So, some services don't have maintainers?

exelib 4y ago

Off-topic question: What are 70 Mio. code lines for? I mean, yeah, there are some services like Uber, Uber eats, but gosh, 70 Mio.? I have worked on very sophisticated products with a bit more than 1 Mio. LOC... Do I miss something in Uber apps and services?

Beltalowda 4y ago
Or "4000+ services", and "500+ Web Apps", and the many thousands of devs they employ. I've wondered for quite a while where all this code and complexity is at. My guess is a lot of it is in their self-driving cars projects and stuff like this, but I don't really know.

AJRF 4y ago

Anyone from Uber here that works on this? You ever expect a DevPod for iOS developers? I've been toying with idea of doing this in work for our devs on Intel MacBooks, using -> https://github.com/sickcodes/Docker-OSX https://github.com/sickcodes/Docker-OSX, which enables local running on iOS devices. I think it's hampered by licensing though, maybe that's why it's not used or mentioned in this post.

chris-laffra 4y ago
I worked on this at Uber in the past and you explain very well why there is no iOS flavor for devpods.
debosmit 4y ago
We’ve thought about this quite deeply at DevZero. Yes, the iOS developer experience can be quite hampered on the local env and requires users to frequently get beefy local laptops/machines. I saw this first hand at Uber from when crowdstrike was rolled out broadly. We have support for AWS-based Mac VMs on DevZero but we don’t find our customers having their biggest issues related to iOS dev yet (we also target enterprise cos that have a vast diversity of tools, mostly backend and front end)

franole 4y ago

I was getting a 404 error. Apparently, because the site redirects me to a non-existent localized version of the page. You can bypass this by forcing English in the URL https://www.uber.com/en-US/blog/devpod-improving-developer-productivity-at-uber/ https://www.uber.com/en-US/blog/devpod-improving-developer-p...

juancn 4y ago

These things usually suck when latency is high... even if you are supporting most major regions it's not usual to find yourself in a situation when you have a few hundred ms latency. Anything over ~40ms is very noticeable when typing and editing. Then you're screwed.

SergeAx 4y ago

And this, ladies and gentlemen, is where monorepos will inevitably take you. There's actually an elementary litmus test on migrating to monorepos. It is a questionnaire with only one question: is your company Google? If the answer is "no" — you don't need a monorepo. You are welcome.