6 ms·

It's always so enlightening to have articles like this one shed light on how companies at scale operate. It goes without saying that many of the problems Stripe

by p-o 2y ago

It's always so enlightening to have articles like this one shed light on how companies at scale operate. It goes without saying that many of the problems Stripe faced with their monorepo isn't application to smaller businesses, but there are still bits and pieces that are applicable to many of us. I've been working on an ephemeral/preview environment operator for Kubernetes(https://github.com/pier-oliviert/sequencer https://github.com/pier-oliviert/sequencer) and as I could agree to a lot of things OP said. I think dev boxes is really the way to go, specially with all the components that makes an application nowadays. But the latency/synchronization issue is a hard topic and it's full of tradeoff. A developer's laptop always ends up being a bespoke environment (yes, Nix/Docker can help with that), and so, there's always a confidence boost when you get your changes up on a standalone environment. It gives you the proof that "hey things are working like I expected them to".

draw_down 2y ago
Right, dev boxes do not need to do double duty as a personal computer plus development target, which allows them to more closely resemble the machine your code will actually run on. They also can be replaced easily, which can be helpful if you ever suspect something is wrong with the box itself - if the new one acts the same way, it wasn't the dev box. I don't recall latency being a big problem in practice. In an organization like this, it's best to keep branches up to date with respect to master anyway, so the diffs from switching between branches should be small. There was a lot of work done to make all this quite performant and nice to use. The slowest part was always CI.
- tmpz22 2y ago
  I feel like we're not getting the right lessons from this. It feels like we're focusing on HOW we can do something versus pausing for a brief moment to consider if we SHOULD in the first place. To me the root issue is the complexity of production environments has expanded to the point of impacting complexity in developer environments just to deploy or test - this is in conjunction with expanding complexity of developer environments just to develop - i.e. web pack. For very large well resourced organizations like Stripe that actually operate at scale that complexity may very well be unavoidable. But most organizations are not Stripe. They should consider decreasing complexity instead of investing in complex tooling to wrangle it. I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding.
  - epinephrinios 2y ago
    Absolutely, I worked on tech behemoths and smaller companies. The dev experience was significantly better when all development was local. I even worked on initiatives to move development away from the cloud, and although other devs were skeptical, they ended up loving it.
  - draw_down 2y ago
    [dead]
  - Eridrus 2y ago
    I think we don't have good solutions for scaling down prod. Our relatively simple prod architecture has 5 containers & a hosted database (so 6 containers when run locally), and any less would impact our product goals. I still find running prod locally valuable, and is the most common way anyone does development here, but containers are fairly heavyweight when you want to run everything on one machine. It's also impossible if you have parts that need special accelerators to get good latency, etc. If you're willing to build everything from scratch, you can have a framework that seamlessly lets you build conceptual services and then separate the physical deployment concerns, like Google has and sometimes even uses. But for the rest of us where we're clobbering together a bunch of different technologies, that's a luxury we can't really afford.
  - jrochkind1 2y ago
    > I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding. I'm not sure "monorepo" means the same thing to you as it does to me? To me, it just means "keep all the code in one repo, instead of trying to split things up into different repos." To me, it's the thing that is the simple solution, it just means "a repo" -- the reason it gets a name is because it's unusual for large orgs with enormous codebases to have everything in one repo, it's unusual for them to do the simple thing that works fine for a small org with a normal codebase. What is it you're suggesting a simple organization should do instead of a "monorepo"?
    - ahtihn 2y ago
hamandcheese 2y ago
My main gripe with the dev box approach is that a cloud instance with similar compute resources as a developers MacBook is hella expensive. Even ignoring compute, a 1TB ebs volume with equivalent performance to a MacBook will probably cost more than the MacBook every month.
- deleted 2y ago
  [deleted]
- axus 2y ago
  The article didn't actually say what "Stripe's cloud environment" was, besides "outside of the production environment". I assumed the company had their own hardware but your assumption is more probable.
- MetaWhirledPeas 2y ago
  Wouldn't this be a reasonable alternative? Asking because I don't have experience with this. 1. New shared builds update container images for applications that comprise the environment 2. Rather than a "devbox", devs use something like Docker Compose to utilize the images locally. Presumably this would be configured identically to the proposed devbox, except with something like a volume pointing to local code. I'm interested in learning more about this. It seems like a way to get things done locally without involving too many cloud services. Is this how most people do it?
  - DiggyJohnson 2y ago
    I manage a dev environment for a small, inexperienced (but eager) team and I have a similar setup. I’ll do a write up at some point if I have time. It can work, and does for me, but there are some funny consequences can end up mediate the relationship between a developer’s computer and his code, which is a terrible place to be.
  - physicles 2y ago
    We do this, but with k3s instead of docker compose (it’s a wonderful single-box substitute for full k8s), and a developer starts by building the relevant container images locally. If everything works, it takes about 3 minutes to get to a working environment with about a dozen services. We steer clear of proprietary cloud services. In place of S3, we use minio. The one sore spot I haven’t been able to solve yet is interactive debugging. k8s pushes you toward a model where all services need a pile of environment variables to run, so setting these up is pretty painful. In practice, though, all rapid iteration happens inside unit tests, so not having interactive debugging isn’t much of a productivity drag.
closeparen 2y ago
I find the devbox approach very frustrating because the JetBrains IDEs are leaps and bounds ahead of everything else in terms of code intelligence, but only work well locally. VSCode is very slightly more capable than plain text editor + sync or terminal-based editor over SSH, but only slightly. It's darkly amusing how we have all these black-magic LLM coding assistants but we can't be reasonably assured of even 2000s level type-aware autocomplete.
- olejorgenb 2y ago
  > VSCode is very slightly more capable than plain text editor What? For which languages are you talking about? For python, VSCode is leaps and bounds ahead of PyCharm if your project is well typed. JetBrains offer a remote solution now though: https://www.jetbrains.com/remote-development/gateway/ https://www.jetbrains.com/remote-development/gateway/
  - closeparen 2y ago
    I work in Go. My company keeps trying to push us onto their VSCode based remote environment. “Find all references” doesn’t, “go to definition” works maybe 30% of the time, and the Go LSP daemon needs to force killed dozens of times in a working session, taking several minutes to recover each time. The autocomplete suggestions are about 3x as likely to be from the VSCode fuzzy matching thing or Copilot slop as actually existing symbols that type check in context. JetBrains Projector and Gateway meanwhile lock up or outright crash several times an hour; text input and scrolling are not smooth.

by p-o 2y ago

draw_down 2y ago

Right, dev boxes do not need to do double duty as a personal computer plus development target, which allows them to more closely resemble the machine your code will actually run on. They also can be replaced easily, which can be helpful if you ever suspect something is wrong with the box itself - if the new one acts the same way, it wasn't the dev box. I don't recall latency being a big problem in practice. In an organization like this, it's best to keep branches up to date with respect to master anyway, so the diffs from switching between branches should be small. There was a lot of work done to make all this quite performant and nice to use. The slowest part was always CI.

tmpz22 2y ago
I feel like we're not getting the right lessons from this. It feels like we're focusing on HOW we can do something versus pausing for a brief moment to consider if we SHOULD in the first place. To me the root issue is the complexity of production environments has expanded to the point of impacting complexity in developer environments just to deploy or test - this is in conjunction with expanding complexity of developer environments just to develop - i.e. web pack. For very large well resourced organizations like Stripe that actually operate at scale that complexity may very well be unavoidable. But most organizations are not Stripe. They should consider decreasing complexity instead of investing in complex tooling to wrangle it. I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding.
- epinephrinios 2y ago
  Absolutely, I worked on tech behemoths and smaller companies. The dev experience was significantly better when all development was local. I even worked on initiatives to move development away from the cloud, and although other devs were skeptical, they ended up loving it.
- draw_down 2y ago
  [dead]
- Eridrus 2y ago
  I think we don't have good solutions for scaling down prod. Our relatively simple prod architecture has 5 containers & a hosted database (so 6 containers when run locally), and any less would impact our product goals. I still find running prod locally valuable, and is the most common way anyone does development here, but containers are fairly heavyweight when you want to run everything on one machine. It's also impossible if you have parts that need special accelerators to get good latency, etc. If you're willing to build everything from scratch, you can have a framework that seamlessly lets you build conceptual services and then separate the physical deployment concerns, like Google has and sometimes even uses. But for the rest of us where we're clobbering together a bunch of different technologies, that's a luxury we can't really afford.
- jrochkind1 2y ago
  > I'd go as far as to suggest both monorepos and dev-boxes are complex toolchains that many organizations should consider avoiding. I'm not sure "monorepo" means the same thing to you as it does to me? To me, it just means "keep all the code in one repo, instead of trying to split things up into different repos." To me, it's the thing that is the simple solution, it just means "a repo" -- the reason it gets a name is because it's unusual for large orgs with enormous codebases to have everything in one repo, it's unusual for them to do the simple thing that works fine for a small org with a normal codebase. What is it you're suggesting a simple organization should do instead of a "monorepo"?
  - ahtihn 2y ago

hamandcheese 2y ago

My main gripe with the dev box approach is that a cloud instance with similar compute resources as a developers MacBook is hella expensive. Even ignoring compute, a 1TB ebs volume with equivalent performance to a MacBook will probably cost more than the MacBook every month.

deleted 2y ago
[deleted]
axus 2y ago
The article didn't actually say what "Stripe's cloud environment" was, besides "outside of the production environment". I assumed the company had their own hardware but your assumption is more probable.
MetaWhirledPeas 2y ago
Wouldn't this be a reasonable alternative? Asking because I don't have experience with this. 1. New shared builds update container images for applications that comprise the environment 2. Rather than a "devbox", devs use something like Docker Compose to utilize the images locally. Presumably this would be configured identically to the proposed devbox, except with something like a volume pointing to local code. I'm interested in learning more about this. It seems like a way to get things done locally without involving too many cloud services. Is this how most people do it?
- DiggyJohnson 2y ago
  I manage a dev environment for a small, inexperienced (but eager) team and I have a similar setup. I’ll do a write up at some point if I have time. It can work, and does for me, but there are some funny consequences can end up mediate the relationship between a developer’s computer and his code, which is a terrible place to be.
- physicles 2y ago
  We do this, but with k3s instead of docker compose (it’s a wonderful single-box substitute for full k8s), and a developer starts by building the relevant container images locally. If everything works, it takes about 3 minutes to get to a working environment with about a dozen services. We steer clear of proprietary cloud services. In place of S3, we use minio. The one sore spot I haven’t been able to solve yet is interactive debugging. k8s pushes you toward a model where all services need a pile of environment variables to run, so setting these up is pretty painful. In practice, though, all rapid iteration happens inside unit tests, so not having interactive debugging isn’t much of a productivity drag.

closeparen 2y ago

I find the devbox approach very frustrating because the JetBrains IDEs are leaps and bounds ahead of everything else in terms of code intelligence, but only work well locally. VSCode is very slightly more capable than plain text editor + sync or terminal-based editor over SSH, but only slightly. It's darkly amusing how we have all these black-magic LLM coding assistants but we can't be reasonably assured of even 2000s level type-aware autocomplete.

olejorgenb 2y ago
> VSCode is very slightly more capable than plain text editor What? For which languages are you talking about? For python, VSCode is leaps and bounds ahead of PyCharm if your project is well typed. JetBrains offer a remote solution now though: https://www.jetbrains.com/remote-development/gateway/ https://www.jetbrains.com/remote-development/gateway/
- closeparen 2y ago
  I work in Go. My company keeps trying to push us onto their VSCode based remote environment. “Find all references” doesn’t, “go to definition” works maybe 30% of the time, and the Go LSP daemon needs to force killed dozens of times in a working session, taking several minutes to recover each time. The autocomplete suggestions are about 3x as likely to be from the VSCode fuzzy matching thing or Copilot slop as actually existing symbols that type check in context. JetBrains Projector and Gateway meanwhile lock up or outright crash several times an hour; text input and scrolling are not smooth.