7 ms·

We are moving to such cloud environment, and it makes me sick. Maybe you need to dockerise Mongo, MySql and 5 other dependencies - I can get this, but I don't

We are moving to such cloud environment, and it makes me sick. Maybe you need to dockerise Mongo, MySql and 5 other dependencies - I can get this, but I don't get it why the rest the code should still be running in the cloud. Python, Rails, Node? Why? Developers should be able to run 1 shell commands to install node. Dev experience excuses are just excuses for a bad setup. So, fix your setup, please. Not being able to run the project natively is big red flag for me and when I move jobs will be my 1st question.

why5s 4y ago
For work-related stuff: I disagree. We're not even at Uber's scale at my company and I hate having to manage the ever-changing set of dependencies that I have no control over.
- layer8 4y ago
  The thing is, it doesn’t have to be ever-changing, and creating a reproducible working setup locally can be achieved by appropriate setup scripts.
  - paxys 4y ago
    Easy in theory, but always breaks down for a non-trivial project with >5 developers on it.
  - tylerhou 4y ago
    Setup scripts that mutate an operating system are fragile. Break something? Unless you understand how the scripts work (which will become stupidly complex at a scale like Uber’s) you will have to reinstall your machine. Or you will have to staff a ton of support to help users when they do break their configuration. Spinning up a VM with an image containing all the development tools is a much smoother experience most of the time. The only reason why I don’t use it where I work is because I use vim and network adds too much latency for me.
    - Arelius 4y ago
      That's a false dichotomy. Just because you don't have a VM doesn't mean that the alternative is a build environment and setup that mutates an operating system.
    - layer8 4y ago
      Using a VM is fine. One can use it locally.
  - jahewson 4y ago
    In practice setup scripts are brittle. New person joins the team and their script fails because it turns out everyone else’s dev environment was only working due to something left behind by some older script. Hotfix requires checking out an old branch but now I need to run an old script but my setup is from the future - the old script doesn’t know what needs to be un-done to get the system to a consistent state. And what about data? My old data has gone, the data I have is from the future. Never mind simple stuff like the script author assuming node is in /usr/bin but that’s not true for me because I use nvm.
    - layer8 4y ago
      I'm sorry, but this is the task of proper configuration management. Yes, don't depend on local stuff that isn't configuration-managed. Don't have a workflow where you check out an old branch in a newer environment. Of course you need a way to establish the older environment in that case. I'm assuming that Devpod does a similar thing on the server side. My point is, the ability to reproduce a working setup doesn't imply a requirement of having to work remotely.
      - unity1001 4y ago
        > I'm assuming that Devpod does a similar thing on the server side That's why they use devpods. You moved entire configuration to the cloud. There is absolutely no reason why it should be run in your local environment.
    - mplanchard 4y ago
      We’ve had good luck using nix for this. Same dependencies for the local dev environment as for the built containers. Same config. All deterministic. And not just major deps like programming languages, but sed, bash, grep, and all the other shell tools, so no more worrying about people running scripts on mac’s ancient version of bash.
  - lhorie 4y ago
    One of the reasons we developed devpods was because our setup scripts were unmanageable. Instructions to new employees would say things like "run this thing, scroll up past dozens of pages of stdout noise and manually deal w/ the errors buried therein by looking up relevant FAQs in some doc somewhere" The scripts would touch every technology imaginable, from brew to npm to arc (phabricator's cli) to proprietary tools and no single person understands how setup scripts work in their entirety. One exercise we'd get new employees to run through was get them to brainstorm about how some system ought to work. The lesson was that just about any idea they could come up with would have already been tried (and failed). I'm told that devpods aren't even the first time we tried cloud dev envs. Presumably lots of lessons were learned from previous attempts at improving dev envs.
    - dpiers 4y ago
      Long before devpods (circa 2014) we had boxer, which was a command line tool that abstracted away the nitty gritty details of setting up a dev environment by orchestrating calls to AWS, vagrant, puppet, rsync, etc for you. However this was also a time when there were only a handful of services you would need to run, and because remote editing tools at the time weren’t as nice as they are now, it only worked really well if your preferred editor was vim/emacs so you didn’t have to deal with the (occasionally buggy) remote file syncing. The devpod flow is a lot smoother. I had my laptop replaced recently and was up it running again in an amount of time that felt like cheating.
    - yjftsjthsd-h 4y ago
      > run this thing, scroll up past dozens of pages of stdout noise and manually deal w/ the errors buried therein by looking up relevant FAQs in some doc somewhere Okay, but if you can define/script your environment enough to run in a pod, couldn't you just run that locally? You already have to solve the manual steps either way...
      - debosmit 4y ago
        Re: if a computer can run scripts in a pod, can’t you just run them locally? It’s a construct of the order of operations. One core issue with local is the variety of OS’ and local build tools that would fundamentally mess with the centralized scripts. Getting company-wide setup scripts to work on top of existing laptop config was a continuous challenge. Hence, having a consistent baseline (OS flavor, system-level packages) on top of which the company-wide “setup script” is added followed by “developer-customizations” seems to work great. Central teams can manage the first couple of steps and individual user-specific configuration can be managed much better in a decentralized manner.
  - unity1001 4y ago
    > it doesn’t have to be ever-changing It IS ever-changing. One library among the zillions of local dependencies that you need to build something changes, and you have to go through the dependency hell. If entire software world valued backwards compatibility and vigilantly guarded it, that wouldnt be a problem. But in the package hell that we are living in today, every other day a package update brings some incompatibility or breaking change for this or that other thing.
  - rtpg 4y ago
    But now you're saying "we shouldn't change dependencies". Pypy turns out to work nicer than CPython? now you have a rollout project to your team. Swapping out some libraries? Another rollout project. Now, it's great if you can avoid complex setups in general cuz complex is harder no matter what! But if you're starting from a complex setup, having easy ways to roll out changes is an important step in actually doing the simplification work to get to where you want to be!
0xB31B1B 4y ago
Its a lot easier to manage dev envs if everyone is literally using cloned ec2 instances that can be remotely fixed/managed by devexp teams. These type of projects are a big step forward IMO
- nithayakumar 4y ago
  Thats definitely one way to do it. Im a founder at a remote dev infra startup (usenimbus.com) and this was the model we started with because it was just so simple. But we quickly learned theres no one size fit all solution so expanded into supporting containers, terraform, etc.
allset_ 4y ago
> Why? Three reasons from my perspective: 1) There's no setup steps. You just open your editor of choice and everything is set up for you. All the build tools, linters, specific versions of software $XYZ, etc. 2) Large VM (16 core, 96GB of RAM in my case) speeds builds and tests up dramatically. 3) Zero productivity lost if your laptop breaks. Just grab a new one from IT and you're up and running exactly where you left off with zero effort. > Not being able to run the project natively What do you mean by this? It's just running on a remote server rather than your laptop.
- calvinmorrison 4y ago
  Yeah as ops gets more complex i don't want to tell developers they need to install xyz with a hundred specific versions. Just grab the latest app image
- eikenberry 4y ago
  I don't see #1 happening. There is no way anyone has my personally customized development setup in some VM ready to use. More likely it is going to be one common setup that they want everyone to use.
  - WaxProlix 4y ago
    It's not for you, it's for every new hire. You can always build on top of the base image, but it sets a nice lower bound and gets people started fast.
    - eikenberry 4y ago
      Problem is with people who put together these sorts of systems tend to think theirs is best and everyone should use it. For example I once interviewed at a place where every developer got an identical Mac with the dev environment already setup and you couldn't use something else as they thought it better if everyone had the same setup to enable easier pairing. I could really see something similar happening with these systems.. seems like the kind of setup that would have appealed to that company I interviewed at.
__MatrixMan__ 4y ago
Also, It just passes the problem down the line. If your dependencies are too messy to manage locally as a dev then they're gonna be an ops nightmare too. Just commit to maintaining a flake.nix and use whatever computer you want.
berkle4455 4y ago
I'm 100% the opposite side of this argument. Running your entire stack locally is a silly trend that cost us a decade of productivity. In the early to late 2000's the remote-dev approach was very common. It wasn't "push a button and you have a dev instance!" easy but it yielded similar results.
- aaomidi 4y ago
  I'm slowly getting to the point of "I want my computers to be a thin client around some config files". Treat your computer like cattle, rather than a pet, etc etc.
  - intelVISA 4y ago
    [Obligatory Nix stanning here]
    - pxc 4y ago
      oblig. advice to wipe / on every boot: https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings
      - tome 4y ago
        Great idea! Also, restore from backup on every boot. The trouble is, I never reboot.
  - eikenberry 4y ago
    Same. I'm considering using a flatpak for this as it seems to have an interesting feature set to build a pre-setup development environment on top of. Just want my shell setup, tooling configs and a few other things bundled up in an easy to use fashion. Currently I just have a git repo with my setup mostly in it (sans executables) with a way to get it going on a new machine. It works but is rather hackish and requires a bit of work to keep in sync.
- raverbashing 4y ago
malux85 4y ago
For me it’s about the hardware, I do molecular simulations and I have a machine with 96 cores and 512GB of RAM and 4 GPUs in it - I wouldn’t want this noisy machine in my house so using VSCode with remote server allows for everything I need to do - interactive debugging just like it’s local, automatic ssh means the terminal feels like it’s local, only a slight (few hundred millisecond) delay on saving, It really feels like it’s local, I enjoy it
lopkeny12ko 4y ago
Same here. We have a similar setup as described in this article. But they also issued me a 16-core workstation with 64 GB of RAM! Like just let me use my hardware goddammit.
- eloisius 4y ago
  In a large org those local resources are allocated for running Tanium.
guusmaan 4y ago
I think you’re overlooking the beginning of this article where they specifically call out the use-case is moving towards their monorepo(s). Do you really think all of Uber’s source could be built and rebuilt continuously on one developer laptop while keeping pace with organization-wide releases? It’s one thing to build a sub module or a few modules locally, but they’re specifically talking about the efficiencies gained through caching ALL build artifacts, source code, and leveraging data locality to place an otherwise inordinate amount of compute power in the hands of developers to do as they please.
armchairhacker 4y ago
It's one thing to "develop" in vim over a slow ssh session or VNC. The latency makes that very annoying. But we're at the point where we can still have a lot of analysis on the machine, and offload the slow analysis and building onto a more powerful computer, so that remote development is faster even with latency. And also getting to the point where internet connectivity is really fast so latency is low. Like with most systems, the important part of remote development is that it's done well. And it seems like most employees are comfortable with Uber's setup. Although, I do have to say that most companies really shouldn't use remote development, only if they have some excuse like they're Uber sized so that the benefits outweigh the costs. I've done remote work at uni, and they're servers and integrations aren't nearly as good so it's a chore; it's faster to use Mutagen and just develop locally then build/deploy remotely.
unity1001 4y ago
> why the rest the code should still be running in the cloud. Python, Rails, Node? Why? Developers should be able to run 1 shell commands to install node. Because, making local environments exactly identical to the actual production environment is nigh on impossible. There still will be minor differences. And, to maintain the local development environment, crap ton of work will go to however that environment is maintained. If engineers are maintaining it themselves, each of them will literally waste time on maintaining the local dependencies needed for the local environment - frequently encountering blockers due to package management hell that we are living in these days breaking one thing or the other. If you have 100 engineers as an example, your organization will lose 100 man-hours each month to such local development environment issues. If you go the route of having infra or dev experience teams etc maintain them, then that team will be spending that effort to keep the scripts and whatever being used to keep the remote local environments in the engineers' computers up to date and working. Instead, that infra team can just prop up dev versions of their infra/cluster/whatever, give the engineers access to that environment through a vpn etc, and voila - you instantly removed a lot of that lost man-hours. Moreover, you will not never encounter any totally unexpected bug or performance problems that could end up coming to being from there being unforeseen incompatibilities in between local environments of the engineers and the actual prod environment. > So, fix your setup Life is not long enough for hundreds of engineers being spending their time on fixing totally unnecessary package management conflicts that are created by the utterly insufferable package and dependency hell that we are living in today. If you like suffering through that dependency hell, good for you. Most of us prefer to ship code and make things happen.
throwawaaarrgh 4y ago
How do you run a job that requires 64GB of RAM and dies when you run out of memory? Or is very CPU intense and takes 3 days on your laptop? If you test it in CI, how long is the dev cycle waiting for your CI to run? What if every dev ends up with slightly different local env that creates different build deps, that turn into test differences? How do you connect the cloud stuff to the non-cloud stuff without running into yet more complexity trying to connect them together?

why5s 4y ago

For work-related stuff: I disagree. We're not even at Uber's scale at my company and I hate having to manage the ever-changing set of dependencies that I have no control over.

layer8 4y ago
The thing is, it doesn’t have to be ever-changing, and creating a reproducible working setup locally can be achieved by appropriate setup scripts.
- paxys 4y ago
  Easy in theory, but always breaks down for a non-trivial project with >5 developers on it.
- tylerhou 4y ago
  Setup scripts that mutate an operating system are fragile. Break something? Unless you understand how the scripts work (which will become stupidly complex at a scale like Uber’s) you will have to reinstall your machine. Or you will have to staff a ton of support to help users when they do break their configuration. Spinning up a VM with an image containing all the development tools is a much smoother experience most of the time. The only reason why I don’t use it where I work is because I use vim and network adds too much latency for me.
  - Arelius 4y ago
    That's a false dichotomy. Just because you don't have a VM doesn't mean that the alternative is a build environment and setup that mutates an operating system.
  - layer8 4y ago
    Using a VM is fine. One can use it locally.
- jahewson 4y ago
  In practice setup scripts are brittle. New person joins the team and their script fails because it turns out everyone else’s dev environment was only working due to something left behind by some older script. Hotfix requires checking out an old branch but now I need to run an old script but my setup is from the future - the old script doesn’t know what needs to be un-done to get the system to a consistent state. And what about data? My old data has gone, the data I have is from the future. Never mind simple stuff like the script author assuming node is in /usr/bin but that’s not true for me because I use nvm.
  - layer8 4y ago
    I'm sorry, but this is the task of proper configuration management. Yes, don't depend on local stuff that isn't configuration-managed. Don't have a workflow where you check out an old branch in a newer environment. Of course you need a way to establish the older environment in that case. I'm assuming that Devpod does a similar thing on the server side. My point is, the ability to reproduce a working setup doesn't imply a requirement of having to work remotely.
    - unity1001 4y ago
      > I'm assuming that Devpod does a similar thing on the server side That's why they use devpods. You moved entire configuration to the cloud. There is absolutely no reason why it should be run in your local environment.
  - mplanchard 4y ago
    We’ve had good luck using nix for this. Same dependencies for the local dev environment as for the built containers. Same config. All deterministic. And not just major deps like programming languages, but sed, bash, grep, and all the other shell tools, so no more worrying about people running scripts on mac’s ancient version of bash.
- lhorie 4y ago
  One of the reasons we developed devpods was because our setup scripts were unmanageable. Instructions to new employees would say things like "run this thing, scroll up past dozens of pages of stdout noise and manually deal w/ the errors buried therein by looking up relevant FAQs in some doc somewhere" The scripts would touch every technology imaginable, from brew to npm to arc (phabricator's cli) to proprietary tools and no single person understands how setup scripts work in their entirety. One exercise we'd get new employees to run through was get them to brainstorm about how some system ought to work. The lesson was that just about any idea they could come up with would have already been tried (and failed). I'm told that devpods aren't even the first time we tried cloud dev envs. Presumably lots of lessons were learned from previous attempts at improving dev envs.
  - dpiers 4y ago
    Long before devpods (circa 2014) we had boxer, which was a command line tool that abstracted away the nitty gritty details of setting up a dev environment by orchestrating calls to AWS, vagrant, puppet, rsync, etc for you. However this was also a time when there were only a handful of services you would need to run, and because remote editing tools at the time weren’t as nice as they are now, it only worked really well if your preferred editor was vim/emacs so you didn’t have to deal with the (occasionally buggy) remote file syncing. The devpod flow is a lot smoother. I had my laptop replaced recently and was up it running again in an amount of time that felt like cheating.
  - yjftsjthsd-h 4y ago
    > run this thing, scroll up past dozens of pages of stdout noise and manually deal w/ the errors buried therein by looking up relevant FAQs in some doc somewhere Okay, but if you can define/script your environment enough to run in a pod, couldn't you just run that locally? You already have to solve the manual steps either way...
    - debosmit 4y ago
      Re: if a computer can run scripts in a pod, can’t you just run them locally? It’s a construct of the order of operations. One core issue with local is the variety of OS’ and local build tools that would fundamentally mess with the centralized scripts. Getting company-wide setup scripts to work on top of existing laptop config was a continuous challenge. Hence, having a consistent baseline (OS flavor, system-level packages) on top of which the company-wide “setup script” is added followed by “developer-customizations” seems to work great. Central teams can manage the first couple of steps and individual user-specific configuration can be managed much better in a decentralized manner.
- unity1001 4y ago
  > it doesn’t have to be ever-changing It IS ever-changing. One library among the zillions of local dependencies that you need to build something changes, and you have to go through the dependency hell. If entire software world valued backwards compatibility and vigilantly guarded it, that wouldnt be a problem. But in the package hell that we are living in today, every other day a package update brings some incompatibility or breaking change for this or that other thing.
- rtpg 4y ago
  But now you're saying "we shouldn't change dependencies". Pypy turns out to work nicer than CPython? now you have a rollout project to your team. Swapping out some libraries? Another rollout project. Now, it's great if you can avoid complex setups in general cuz complex is harder no matter what! But if you're starting from a complex setup, having easy ways to roll out changes is an important step in actually doing the simplification work to get to where you want to be!

0xB31B1B 4y ago

Its a lot easier to manage dev envs if everyone is literally using cloned ec2 instances that can be remotely fixed/managed by devexp teams. These type of projects are a big step forward IMO

nithayakumar 4y ago
Thats definitely one way to do it. Im a founder at a remote dev infra startup (usenimbus.com) and this was the model we started with because it was just so simple. But we quickly learned theres no one size fit all solution so expanded into supporting containers, terraform, etc.

allset_ 4y ago

> Why? Three reasons from my perspective: 1) There's no setup steps. You just open your editor of choice and everything is set up for you. All the build tools, linters, specific versions of software $XYZ, etc. 2) Large VM (16 core, 96GB of RAM in my case) speeds builds and tests up dramatically. 3) Zero productivity lost if your laptop breaks. Just grab a new one from IT and you're up and running exactly where you left off with zero effort. > Not being able to run the project natively What do you mean by this? It's just running on a remote server rather than your laptop.

calvinmorrison 4y ago
Yeah as ops gets more complex i don't want to tell developers they need to install xyz with a hundred specific versions. Just grab the latest app image
eikenberry 4y ago
I don't see #1 happening. There is no way anyone has my personally customized development setup in some VM ready to use. More likely it is going to be one common setup that they want everyone to use.
- WaxProlix 4y ago
  It's not for you, it's for every new hire. You can always build on top of the base image, but it sets a nice lower bound and gets people started fast.
  - eikenberry 4y ago
    Problem is with people who put together these sorts of systems tend to think theirs is best and everyone should use it. For example I once interviewed at a place where every developer got an identical Mac with the dev environment already setup and you couldn't use something else as they thought it better if everyone had the same setup to enable easier pairing. I could really see something similar happening with these systems.. seems like the kind of setup that would have appealed to that company I interviewed at.

__MatrixMan__ 4y ago

Also, It just passes the problem down the line. If your dependencies are too messy to manage locally as a dev then they're gonna be an ops nightmare too. Just commit to maintaining a flake.nix and use whatever computer you want.

berkle4455 4y ago

I'm 100% the opposite side of this argument. Running your entire stack locally is a silly trend that cost us a decade of productivity. In the early to late 2000's the remote-dev approach was very common. It wasn't "push a button and you have a dev instance!" easy but it yielded similar results.

aaomidi 4y ago
I'm slowly getting to the point of "I want my computers to be a thin client around some config files". Treat your computer like cattle, rather than a pet, etc etc.
- intelVISA 4y ago
  [Obligatory Nix stanning here]
  - pxc 4y ago
    oblig. advice to wipe / on every boot: https://grahamc.com/blog/erase-your-darlings https://grahamc.com/blog/erase-your-darlings
    - tome 4y ago
      Great idea! Also, restore from backup on every boot. The trouble is, I never reboot.
- eikenberry 4y ago
  Same. I'm considering using a flatpak for this as it seems to have an interesting feature set to build a pre-setup development environment on top of. Just want my shell setup, tooling configs and a few other things bundled up in an easy to use fashion. Currently I just have a git repo with my setup mostly in it (sans executables) with a way to get it going on a new machine. It works but is rather hackish and requires a bit of work to keep in sync.
raverbashing 4y ago

malux85 4y ago

For me it’s about the hardware, I do molecular simulations and I have a machine with 96 cores and 512GB of RAM and 4 GPUs in it - I wouldn’t want this noisy machine in my house so using VSCode with remote server allows for everything I need to do - interactive debugging just like it’s local, automatic ssh means the terminal feels like it’s local, only a slight (few hundred millisecond) delay on saving, It really feels like it’s local, I enjoy it

lopkeny12ko 4y ago

Same here. We have a similar setup as described in this article. But they also issued me a 16-core workstation with 64 GB of RAM! Like just let me use my hardware goddammit.

eloisius 4y ago
In a large org those local resources are allocated for running Tanium.

guusmaan 4y ago

I think you’re overlooking the beginning of this article where they specifically call out the use-case is moving towards their monorepo(s). Do you really think all of Uber’s source could be built and rebuilt continuously on one developer laptop while keeping pace with organization-wide releases? It’s one thing to build a sub module or a few modules locally, but they’re specifically talking about the efficiencies gained through caching ALL build artifacts, source code, and leveraging data locality to place an otherwise inordinate amount of compute power in the hands of developers to do as they please.

armchairhacker 4y ago

It's one thing to "develop" in vim over a slow ssh session or VNC. The latency makes that very annoying. But we're at the point where we can still have a lot of analysis on the machine, and offload the slow analysis and building onto a more powerful computer, so that remote development is faster even with latency. And also getting to the point where internet connectivity is really fast so latency is low. Like with most systems, the important part of remote development is that it's done well. And it seems like most employees are comfortable with Uber's setup. Although, I do have to say that most companies really shouldn't use remote development, only if they have some excuse like they're Uber sized so that the benefits outweigh the costs. I've done remote work at uni, and they're servers and integrations aren't nearly as good so it's a chore; it's faster to use Mutagen and just develop locally then build/deploy remotely.

unity1001 4y ago

> why the rest the code should still be running in the cloud. Python, Rails, Node? Why? Developers should be able to run 1 shell commands to install node. Because, making local environments exactly identical to the actual production environment is nigh on impossible. There still will be minor differences. And, to maintain the local development environment, crap ton of work will go to however that environment is maintained. If engineers are maintaining it themselves, each of them will literally waste time on maintaining the local dependencies needed for the local environment - frequently encountering blockers due to package management hell that we are living in these days breaking one thing or the other. If you have 100 engineers as an example, your organization will lose 100 man-hours each month to such local development environment issues. If you go the route of having infra or dev experience teams etc maintain them, then that team will be spending that effort to keep the scripts and whatever being used to keep the remote local environments in the engineers' computers up to date and working. Instead, that infra team can just prop up dev versions of their infra/cluster/whatever, give the engineers access to that environment through a vpn etc, and voila - you instantly removed a lot of that lost man-hours. Moreover, you will not never encounter any totally unexpected bug or performance problems that could end up coming to being from there being unforeseen incompatibilities in between local environments of the engineers and the actual prod environment. > So, fix your setup Life is not long enough for hundreds of engineers being spending their time on fixing totally unnecessary package management conflicts that are created by the utterly insufferable package and dependency hell that we are living in today. If you like suffering through that dependency hell, good for you. Most of us prefer to ship code and make things happen.

throwawaaarrgh 4y ago

How do you run a job that requires 64GB of RAM and dies when you run out of memory? Or is very CPU intense and takes 3 days on your laptop? If you test it in CI, how long is the dev cycle waiting for your CI to run? What if every dev ends up with slightly different local env that creates different build deps, that turn into test differences? How do you connect the cloud stuff to the non-cloud stuff without running into yet more complexity trying to connect them together?