12 ms·

Author here, thanks for sharing! This is a strategy I’ve used across many projects in several languages, from big Rails apps to the Go monolith we deploy at in

Author here, thanks for sharing! This is a strategy I’ve used across many projects in several languages, from big Rails apps to the Go monolith we deploy at incident.io today. It’s just one way you can make a monolith much more robust without moving into separate services, which helps you keep the benefits of a monolithic application for longer. Hope people find it interesting.

herodoturtle 3y ago
Thank you for sharing this article! > But having spent half a decade stewarding a Ruby monolith from 20 to 200 engineers and watched its modest 10GB Postgres database grow beyond 5TB, there’s definitely a point where the pain outweighs the benefits. Woah! I for one would also love to hear your insights on scaling personnel from 20 to 200. We’re in a similar boat / anticipated growth phase (at 20 odd engineers), and whilst there’s a lot of content on this topic, I’d appreciate your practical “from the coal face” take (much like what you’ve done with this monolith article). Perhaps a follow up article? ^_^
- lawrjone 3y ago
  Glad you enjoyed the article! The experience I reference was from my time at GoCardless, where I was a Principal SRE leading the team who were on-call for the big Rails monolith. I’ll put the topic of “what does 10x eng team scaling feel like” on my todo list for posts, but if you’re interested there’s a bunch of articles on my personal blog from this time. One that might be interesting is “Growing into Platform Engineering” about how the SRE function evolved: https://blog.lawrencejones.dev/growing-into-platform-engineering/ https://blog.lawrencejones.dev/growing-into-platform-enginee...
rfoo 3y ago
Question: Have you considered one separate binary (but still sharing the entire codebase) for each "workload" instead of using a flag to switch? If so, can you highlight the advantage of the flag way?
- lawrjone 3y ago
  Yep: this is definitely a valid alternative. The reasons we went with one binary are: - We can ensure common runtime setup by only having one codepath that runs the initialisation. It's nice that every deployment definitely instantiates the Prometheus server and initialises secrets from Google Secret Manager at the same point and in the same way. - Compiling three targets adds time to our build process. Even when Go will cache much of the compilation, linking separate binaries is still another ~30s and ~300MB of binary to our resulting docker image. - It's very convenient in development to run all deployments inside of a single binary for hot-reloading purposes. This is how we develop the app locally. I don't think it makes much difference which you choose in terms of the operational elements. But for the reasons above, there are clear advantages to the single binary for our use cases.
  - rfoo 3y ago
    Thanks!
Jupe 3y ago
While I completely agree that the efforts to build and maintain a set of micro-services are often better leveraged by a single monolith (even one that has several "run modes" as you've done here), a few questions inevitably come up: How do you coordinate the efforts of 200 engineers on a single repo/code base? Do engineers frequently get into long/drawn-out merge sessions as common code may be modified by a number of engineers who are all trying to merge around the same time? This is actually one of the reasons I really like GOLANG: "A little copying is better than a little dependency."
- krab 3y ago
  I have worked in a company with a monolith and about ten teams working on it. This is what helped: - Merging was automated (a robot tried to run tests against fresh master and merge only if green). - Deploy was fully automated and limited to working hours. - We added tests for problematic parts. For example static analysis for database migrations to prevent only safe actions in an automated fashion. However, if something goes wrong in some component, you have to revert and stop the deploys for everyone which sucks. I'd say around 8 - 10 deploys per day, it makes sense to start splitting the components or at least not adding new teams to the same monolith.
  - withinboredom 3y ago
    I worked on a monolith with 100+ devs working on it daily. Merging was automated as well for us, and a few unit tests were run on a production env — basically tests that asserted an engineer didn’t do anything stupid like add an infinite loop, or any of the other dozens things engineers had done that caused downtime. Deployment was done by running a script that took a lock, monitored the deployment progress, then released the lock. Sometimes you’d run the script and see people who hadn’t deployed yet, so you’d send them a message and ask if it was good to deploy. Sometimes they would say no if they found a last minute bug, so they would deploy your code for you. Surprisingly, coordination isn’t that difficult. Humans are pretty good at talking to each other.

herodoturtle 3y ago

Thank you for sharing this article! > But having spent half a decade stewarding a Ruby monolith from 20 to 200 engineers and watched its modest 10GB Postgres database grow beyond 5TB, there’s definitely a point where the pain outweighs the benefits. Woah! I for one would also love to hear your insights on scaling personnel from 20 to 200. We’re in a similar boat / anticipated growth phase (at 20 odd engineers), and whilst there’s a lot of content on this topic, I’d appreciate your practical “from the coal face” take (much like what you’ve done with this monolith article). Perhaps a follow up article? ^_^

lawrjone 3y ago
Glad you enjoyed the article! The experience I reference was from my time at GoCardless, where I was a Principal SRE leading the team who were on-call for the big Rails monolith. I’ll put the topic of “what does 10x eng team scaling feel like” on my todo list for posts, but if you’re interested there’s a bunch of articles on my personal blog from this time. One that might be interesting is “Growing into Platform Engineering” about how the SRE function evolved: https://blog.lawrencejones.dev/growing-into-platform-engineering/ https://blog.lawrencejones.dev/growing-into-platform-enginee...

rfoo 3y ago

Question: Have you considered one separate binary (but still sharing the entire codebase) for each "workload" instead of using a flag to switch? If so, can you highlight the advantage of the flag way?

lawrjone 3y ago
Yep: this is definitely a valid alternative. The reasons we went with one binary are: - We can ensure common runtime setup by only having one codepath that runs the initialisation. It's nice that every deployment definitely instantiates the Prometheus server and initialises secrets from Google Secret Manager at the same point and in the same way. - Compiling three targets adds time to our build process. Even when Go will cache much of the compilation, linking separate binaries is still another ~30s and ~300MB of binary to our resulting docker image. - It's very convenient in development to run all deployments inside of a single binary for hot-reloading purposes. This is how we develop the app locally. I don't think it makes much difference which you choose in terms of the operational elements. But for the reasons above, there are clear advantages to the single binary for our use cases.
- rfoo 3y ago
  Thanks!

Jupe 3y ago

While I completely agree that the efforts to build and maintain a set of micro-services are often better leveraged by a single monolith (even one that has several "run modes" as you've done here), a few questions inevitably come up: How do you coordinate the efforts of 200 engineers on a single repo/code base? Do engineers frequently get into long/drawn-out merge sessions as common code may be modified by a number of engineers who are all trying to merge around the same time? This is actually one of the reasons I really like GOLANG: "A little copying is better than a little dependency."

krab 3y ago
I have worked in a company with a monolith and about ten teams working on it. This is what helped: - Merging was automated (a robot tried to run tests against fresh master and merge only if green). - Deploy was fully automated and limited to working hours. - We added tests for problematic parts. For example static analysis for database migrations to prevent only safe actions in an automated fashion. However, if something goes wrong in some component, you have to revert and stop the deploys for everyone which sucks. I'd say around 8 - 10 deploys per day, it makes sense to start splitting the components or at least not adding new teams to the same monolith.
- withinboredom 3y ago
  I worked on a monolith with 100+ devs working on it daily. Merging was automated as well for us, and a few unit tests were run on a production env — basically tests that asserted an engineer didn’t do anything stupid like add an infinite loop, or any of the other dozens things engineers had done that caused downtime. Deployment was done by running a script that took a lock, monitored the deployment progress, then released the lock. Sometimes you’d run the script and see people who hadn’t deployed yet, so you’d send them a message and ask if it was good to deploy. Sometimes they would say no if they found a last minute bug, so they would deploy your code for you. Surprisingly, coordination isn’t that difficult. Humans are pretty good at talking to each other.