5 ms·

Great list. Only thing I’d add for the other enterprise developers out there is to first default to not building a distributed system at all, but rather build a

Great list. Only thing I’d add for the other enterprise developers out there is to first default to not building a distributed system at all, but rather build a much smaller monolith. In 25 years I’ve worked for so many orgs that wanted to build The World’s Most Scalable System for what would maybe be a few hundred concurrent users. Not surprisingly, those projects tend to tank.

golergka 6y ago
That's great advice, as long as different systems in this monolith are loosely coupled both in terms of code and data, and you keep in mind that you will need to move them to separate services later on.
lightlazer 6y ago
That is a great advice. However if scalability must be introduced later on, it can be really hard as there are many features that have been added to the monolith, and refactoring it to become scalable can be a huge task. The conditions where the monolith must be converted to a scalable system should be defined as early as possible.
- herodoturtle 6y ago
  This too is excellent advice. Start off with a monolith if it makes most sense (in terms of simplicity / MVP) but from the start keep future scalability in mind and plan for it. I like this.
  - bovermyer 6y ago
    Unless, of course, you intend to cap users at a specific amount for another reason.
- dasil003 6y ago
  Designing something to be “scalable” before it’s actually established is a recipe for premature optimization at best and a poorly baked SOA with service boundaries that make change incredibly difficult at worst. It’s important to keep mind that SOA is about scaling teams first, code second and not really about throughout per se. A share-nothing web tier plus a couple judiciously applied databases and background job queues can effectively scale a huge proportion of applications without the overhead of a full SOA.
amelius 6y ago
The problem with scaling is that visitor count often goes exponentially. So while in the beginning everything seems calm and quiet, suddenly you can see a massive spike in interest. And if you didn't take the right measures, you might lose all that interest.
- deepGem 6y ago
  Yep, it is still a matter of abandoning the monolith and building a new scalable system in whatever fashion you choose at that time, micro services or whatever. In today's cloud deployments where cost is not a mega concern (as compared to having physical servers in a DC), this approach is very much feasible. You could run two systems in parallel (monolith v/s scalable) and ditch the monolith once you have the necessary cutover logistics in place. Of course, the sooner you do this, the better and it is non trivial to figure out the 'when'
- atleta 6y ago
  That's what startups are seeking, but it's not that often at all. Also, exponential doesn't necessarily mean "happens too fast to react". If the number of your users doubles every year, you still have exponential growth, if you only have paid users you still have a very nice growth of business and you also have time to react. It's indeed pretty rare to see services that crumble under the exponential increase in traffic. Especially if it's not a spike from publicity (I don't know how you would say "being slashdotted" these days). It happens, but it's rare. But because it's usually newsworthy or at least interesting it gets more attention and it will feel much more of a danger than it actually is. Both for software developers and entrepreneurs (or even for the general public) it feels lame. Ha! They should have expected this! But the truth is that most of the time you should not prepare for this, because it's pretty rare, it's not even necessary for success, not even for startup scale success at the beginning. It's pretty easy to see actually: (almost) all of today's successful services provided by startups started as monoliths (or maybe more realistically some kind of SOA, because monolith vs. microservices is really a false dichotomy). It did work a decade ago with weaker hardware, why wouldn't it work now?
- toast0 6y ago
  There's times when you might be able to expect a huge spike, but even then, if the product is early, so the overall design is still pretty fluid, I would try to keep flexible. It's more important to be aware of overload, and provide feedback to users (or potential users) and some form of load shedding. And test that all. For a lot of applications, being able to quickly plop together a second monolith could work to address spikes. Or switching to a bigger machine: Epyc 2-socket systems get you up to 128 cores (256 SMT threads), and I think 8TB of ram. You can do an awful lot with that.
herodoturtle 6y ago
Hear hear. This is really important wisdom being shared. And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred. From my humble experience I've found that it is also relatively easier to migrate an established monolith to a semi-distributed system, than it is to build and scale a distributed system from the ground up whilst at the same time trying to figure out all its kinks before it too becomes an established system.
- amelius 6y ago
  > And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred. That depends on what kind of service you offer, of course.
  - herodoturtle 6y ago
    Haha yes, good point - nothing with video ^_^
- kortex 6y ago
  The trick is to find "fault lines" throughout your monolith - places where it's easy to create separation of concerns and narrow interfaces. Maintain and document these like they are external APIs. Only pass simple data through it (serialized or anything easy to serialize). When the time comes, cleave these chunks off and wrap them in RPC/REST/graphql. It's good form even for monoliths since it makes for an easy interface to test.
  - alexjplant 6y ago
    +1 to you sir. I was just having a discussion this past weekend with some technical friends and noted that a well-architected monolith often lends itself to easy deconstruction such that you get the best of both worlds when they're necessary in your product lifecycle.
  - herodoturtle 6y ago
    Good tip, and I learnt some new terminology too ("fault lines"). I guess the trick when building new monoliths is to identify these fault lines up front - as opposed to hopefully finding them later. Will certainly guide my development going forward, thanks.
iamvnraju 6y ago
And most of the time a distributed system just ends up being a distributed monolith.
hawk_ 6y ago
But our industry has all kinds of perverse incentives to do resumé driven development. How does one get to put paxos on their resumé if they built the correct simple solution?

golergka 6y ago

That's great advice, as long as different systems in this monolith are loosely coupled both in terms of code and data, and you keep in mind that you will need to move them to separate services later on.

lightlazer 6y ago

That is a great advice. However if scalability must be introduced later on, it can be really hard as there are many features that have been added to the monolith, and refactoring it to become scalable can be a huge task. The conditions where the monolith must be converted to a scalable system should be defined as early as possible.

herodoturtle 6y ago
This too is excellent advice. Start off with a monolith if it makes most sense (in terms of simplicity / MVP) but from the start keep future scalability in mind and plan for it. I like this.
- bovermyer 6y ago
  Unless, of course, you intend to cap users at a specific amount for another reason.
dasil003 6y ago
Designing something to be “scalable” before it’s actually established is a recipe for premature optimization at best and a poorly baked SOA with service boundaries that make change incredibly difficult at worst. It’s important to keep mind that SOA is about scaling teams first, code second and not really about throughout per se. A share-nothing web tier plus a couple judiciously applied databases and background job queues can effectively scale a huge proportion of applications without the overhead of a full SOA.

amelius 6y ago

The problem with scaling is that visitor count often goes exponentially. So while in the beginning everything seems calm and quiet, suddenly you can see a massive spike in interest. And if you didn't take the right measures, you might lose all that interest.

deepGem 6y ago
Yep, it is still a matter of abandoning the monolith and building a new scalable system in whatever fashion you choose at that time, micro services or whatever. In today's cloud deployments where cost is not a mega concern (as compared to having physical servers in a DC), this approach is very much feasible. You could run two systems in parallel (monolith v/s scalable) and ditch the monolith once you have the necessary cutover logistics in place. Of course, the sooner you do this, the better and it is non trivial to figure out the 'when'
atleta 6y ago
That's what startups are seeking, but it's not that often at all. Also, exponential doesn't necessarily mean "happens too fast to react". If the number of your users doubles every year, you still have exponential growth, if you only have paid users you still have a very nice growth of business and you also have time to react. It's indeed pretty rare to see services that crumble under the exponential increase in traffic. Especially if it's not a spike from publicity (I don't know how you would say "being slashdotted" these days). It happens, but it's rare. But because it's usually newsworthy or at least interesting it gets more attention and it will feel much more of a danger than it actually is. Both for software developers and entrepreneurs (or even for the general public) it feels lame. Ha! They should have expected this! But the truth is that most of the time you should not prepare for this, because it's pretty rare, it's not even necessary for success, not even for startup scale success at the beginning. It's pretty easy to see actually: (almost) all of today's successful services provided by startups started as monoliths (or maybe more realistically some kind of SOA, because monolith vs. microservices is really a false dichotomy). It did work a decade ago with weaker hardware, why wouldn't it work now?
toast0 6y ago
There's times when you might be able to expect a huge spike, but even then, if the product is early, so the overall design is still pretty fluid, I would try to keep flexible. It's more important to be aware of overload, and provide feedback to users (or potential users) and some form of load shedding. And test that all. For a lot of applications, being able to quickly plop together a second monolith could work to address spikes. Or switching to a bigger machine: Epyc 2-socket systems get you up to 128 cores (256 SMT threads), and I think 8TB of ram. You can do an awful lot with that.

herodoturtle 6y ago

Hear hear. This is really important wisdom being shared. And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred. From my humble experience I've found that it is also relatively easier to migrate an established monolith to a semi-distributed system, than it is to build and scale a distributed system from the ground up whilst at the same time trying to figure out all its kinks before it too becomes an established system.

amelius 6y ago
> And with a single monolith you can comfortably handle several thousand concurrent users, not just several hundred. That depends on what kind of service you offer, of course.
- herodoturtle 6y ago
  Haha yes, good point - nothing with video ^_^
kortex 6y ago
The trick is to find "fault lines" throughout your monolith - places where it's easy to create separation of concerns and narrow interfaces. Maintain and document these like they are external APIs. Only pass simple data through it (serialized or anything easy to serialize). When the time comes, cleave these chunks off and wrap them in RPC/REST/graphql. It's good form even for monoliths since it makes for an easy interface to test.
- alexjplant 6y ago
  +1 to you sir. I was just having a discussion this past weekend with some technical friends and noted that a well-architected monolith often lends itself to easy deconstruction such that you get the best of both worlds when they're necessary in your product lifecycle.
- herodoturtle 6y ago
  Good tip, and I learnt some new terminology too ("fault lines"). I guess the trick when building new monoliths is to identify these fault lines up front - as opposed to hopefully finding them later. Will certainly guide my development going forward, thanks.

iamvnraju 6y ago

And most of the time a distributed system just ends up being a distributed monolith.

hawk_ 6y ago

But our industry has all kinds of perverse incentives to do resumé driven development. How does one get to put paxos on their resumé if they built the correct simple solution?