7 ms·

I always find these posts ironic, because I work at a company where we are often times dealing with the opposite problem; trying to redesign systems that worked

I always find these posts ironic, because I work at a company where we are often times dealing with the opposite problem; trying to redesign systems that worked fine when we were a small startup but have hit scaling issues now that we are many times the size. People are always so quick to comment, “don’t worry about scaling... if you are successful, you can redesign at that point.” However, it isn’t always easy to redesign later. Now people are relying on your systems, and you can’t have any down time. All redesigns have to be able to be implemented in a way that can transition from the previous system without breaking anything. As we often say, “we are changing the tires while driving 90mph down the road”. I wish we had spent more time on thinking about scale when we were smaller. Our new mantra is that everything we make now has to be designed to horizontally scale, even if it is something small right now.

rntksi 6y ago
I have had the same issue, and wishing we thought about that up the road rather than down the road is exactly the feeling when we huddle up for team meetings on optimisation. We could not have any down time for most days of the year.
gfodor 6y ago
Everybody at a company experiencing scaling issues feels the way you do. The argument however is if the past went the way you wish it did, arguably the company you currently work for would not exist, because they would have lacked the knowledge that you have now to know which areas were the ones that would matter. Absent that knowledge, they would have built “scalable” systems that still had bottlenecks they could not yet see and also would have adopted extraneous technology that ultimately would be thrown out as the product discarded the parts that ended up not being critical to the value proposition uncovered. And of course, this whole process would have burned time, energy, and cash, not to mention shifted the nature of hires and the composition of the team itself, such that in sum it may have been a sufficient opportunity cost to jeopardize the whole business.
- cortesoft 6y ago
  Yeah, I know this is true, and we always try to remember to understand that the engineering decisions that were made at the time were made for good reasons given the circumstances we were facing. It helps that I was one of the people who made those decisions at the time, so I know we were making what we thought were the best decisions at the time. Still wish we had made some different ones, though!
  - josephg 6y ago
    Hear hear! I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Spoilers: That never happened - the longer we used mongo, the more expensive migrating became. We never did manage to justify the migration. As far as I know the team is still stuck with it, and paying the price. (If you're reading this and you know who you are, I'm sorry!)
    - rubyist5eva 6y ago
      Never use MongoDB. Ever. It's a rule of thumb that hasn't led me astray yet. http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/ http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...
      - dominotw 6y ago
        > It's a rule of thumb that hasn't led me astray yet. How can you know if your decision is correct if you never use it in the first place. Isn't it like only implementing A version of A/B and concluding that not implementing B version was a good decision.
      - jcpst 6y ago
        Meh. I read the article, and I agree with it. But that is not what you are saying. Don’t use it for relational data. Don’t use it as your primary data store. At the enterprise I work at, I’ve had success delivering features where data science would have some job that aggregates production data and dumps it to kafka. Then we’ll map it and dump it in mongo. Stick a .net core microservice on top for retrieval. Works great, has none of the problems mentioned in that article. Of course, this might not make sense if you’re not big enough to where you need kafka, kubernetes, etc But “Never use it” is certainly not good advice.
louwrentius 6y ago
So, Let's Encrypt just uses a very big piece of iron for their database server, they don't scale horizontally. Stack Overflow also famously buys the biggest hardware they can get and upgrade their database server accordingly. I always wonder if horizontal scaling is really always the right answer, when you can just buy a bigger box (of bigger VM) and spare yourself a lot of issues.
- basilgohar 6y ago
  I am not a proponent of scaling out horizontally as the best solution to all things, but being able to do so comes as a result of design that helps to minimize coupling of services and single points of failure, in general, which are other benefits. But I also tend to think "you can always get a bigger box" can take you pretty far, and much further than most people that only think in terms of AWS realize.
- cortesoft 6y ago
  Up to a certain point, sure, you can vertically scale. I work for a large CDN, with many thousands of servers pushing near line-rate out of our NICs... we certainly can't vertically scale that. There is a limit to how much you can vertically scale. Even for the things we CAN vertically scale, we need to be able to handle hardware failure and OS upgrades without taking any downtime, so you are going to need to be able to horizontally scale enough to handle that.
  - jorams 6y ago
    > I work for a large CDN, with many thousands of servers pushing near line-rate out of our NICs... we certainly can't vertically scale that. A CDN is essentially "horizontal scale as a service". The vast majority of products are very different.
    - hinkley 6y ago
      And the products that aren’t should hire out CDN networks to do that work for them instead of trying to roll their own.
ineedasername 6y ago
That's a bit of survivor bias: Most startups don't go from small to large, or even really medium. Most startups are strapped for resources. Paying an extra engineer, or taking valuable time away from an existing one to implement an extremely scalable infrastructure that might not be used. They should focus on shipping a good product that people want, and getting people to buy it. I can't think of startups that failed because their stack needed to be upgraded for scalability. It's generally not why startups fail. I can find a whole lot who burned through their cash before they had a chance to need something orders of magnitude larger in scale than currently necessary. If it's a minimal cost in time & money? Sure, go for high scalability. But it it's going to reduce your runway significantly then worry about the bigger problem of becoming successful enough to need it in the first place. If a startup is really concerned about avoiding the problems you're encountering at your company they should just take an extra week when planning their infrastructure to map out details of how they would transition to a more scalable infrastructure. Then save that file somewhere and dust it off if they ever need it. And if they need it, they'll have ample investment money coming their way to pay for the transition.
tomlagier 6y ago
My last 3 consulting gigs have all had the same general issue - they went with the "quick to start up", scalable solutions and immediately smacked into a brick wall of complexity. These are 1 to 3 person eng teams that started with serverless functions, NoSQL databases, and event-based architecture and not even 6 months down the line had to hire an expensive consultant to shake out their architecture and point them at something less scalable but more practical to build on at their size. If they didn't have VC, they'd likely die on the spot, drowned before they even had a shot at real product-market fit under the weight of their own accidental complexity. Now, is there a set of common patterns that works well enough for startups to iterate quickly with, that _also_ scales? Probably. Can you be successful with NoSQL et al out of the gate? Almost definitely. Does it make it _much harder_? I have three anecdotal data points that point to "yes". From my work over the last year, I'd suggest "choose boring technology" over "worry about scaling early" 10 out of 10 times. You need something that scales _conceptually_ before it can scale in the cloud, if you don't know what your core business value is or how your product actually works it's really damn hard to build a cloud-native app that anyone can actually iterate on. You're fighting incidental complexity every step of the way. Maybe in 5 or 10 years, when the average developer has a wealth of experience in scaling technologies, when serverless functions have fewer dramatic downsides, when we start seeing out-of-the-box event driven full stack frameworks, when the bleeding sharp edges are sanded off NoSQL and we've figured out a good story around schemas and validation, then it starts to look more attractive for young companies. Until then, though? Stick with the boring stuff. EDIT: Though, re-reading this, I suppose I should acknowledge my own survivorship bias - the companies that figured out their scalable, cloud-based tech stack early have no need for my consulting services...
- sk5t 6y ago
  "Choose Boring Technology": http://boringtechnology.club/ http://boringtechnology.club/
- macspoofing 6y ago
  Great comment. This mirrors the ridiculous 'big data' fad of a few years ago, where startups would deploy Hadoop clusters to crunch through Terabytes (or even Gigabytes!!) of data ... a task that could have been done much better by a well resourced monolithic system that was easy to implement, and easy to maintain and understand and just worked.
kortilla 6y ago
There is no startup in the scaling phase that doesn’t have stuff that needs to be rewritten. If your startup’s product is constantly iterating to find product market fit, it’s a fool’s errand to keep making sure everything is perfectly horizontally scalable in the first place. It’s very rare that the things you expect to be important early on actually are in the product that gets popular. You’re absolutely right to change your mantra now that you’re scaling, but you were right to not do it early on. The difference between life an death in an early startup is quick iteration.
dgb23 6y ago
I wonder what the ratio of failed startups that didn't engineer for scaling early vs those that did is. Having to rewrite parts of your application to match scaling issues seems like a great problem to have.
brightball 6y ago
There's two very different ways to interpret the "don't worry about scaling" stance and I think that's where things go sideways. On the one, people take it as literally "don't worry about it, just make it work" and don't think through simple decisions. On the other, with some basic analysis you can identify where your likely bottlenecks are going to be and make some preparations to dramatically simplify scaling when you need to do it. Sometimes minor architectural decisions, that don't take much time, can lead to a happier long term life. But I 100% agree with designing for horizontally scaling. It's one of the reasons I always try to steer people in the direction of Heroku for new projects, because while it reduces your ops burden and makes your life easier...it also forces you to design for horizontal scalability from day 1 via 12 Factor.
- acdha 6y ago
  That’s how I take this, with a related point of thinking about how you’d replace components when the time comes. Maybe you have a monolith-leaning design but you have tests, a clean API with some idea of ownership, etc. rather than a bunch of code directly making SQL calls, copying the same logic instead of using a function, or touching the shared files so when you do split things out and redesign them the process is safe. (Having flashbacks to the same code in 3 PHP repos, maintained by one person, no two copies identical)
Moto7451 6y ago
At my work we’re one step past where it sounds you’re at. I’ve been around since the very early days so I’m both part of the original problem and the solution. Now our biggest problem is we don’t like to work with the things we built to handle scale. I’m not sure how common that is but I don’t believe we’re alone. A lot of the problem from my point of view was trying to chase trends. We have Kafka in places the article’s author would suggest are inappropriate. We have a lot of “one true systems” that hid horizontally scaling instead of making it easy to build clusters of micro-services. Now we’re riding micro-services pretty hard and that isn’t going well since we have too many mixed architectures in use. I think it’s correct to push out worrying about scale while you seek product market fit. I think that once you do need to scale you really need to put thought into the process. This is difficult if everything is on fire and failing. If it’s not, burn some time to plan. It’ll pay off. Be sure you don’t reinvent things you can cheaply buy. Make sure developers still like their day jobs after the changes are made before making more changes with the same underlying design theories.
- Person5478 6y ago
  As companies grow there will be "pain points" that come up due to scaling (performance, architecture, development, etc). When you get to that point it's normal to wish the pain didn't exist by "doing it right" the first time, but I think it's still better to wait until you have the pain. Because that pain tells you unequivocally what you need to value in the scaling.
bostonsre 6y ago
Yea, I don't think it's always a waste of time to think about scale at the start. You don't have to do all the work up front, but you can definitely put thought into doing the early work in a way that will make scaling easier later. There is no sense in committing to an architecture early that limits your options later. Also, avoiding sloppiness and ensuring simplicity up front usually pays dividends later.
DenisM 6y ago
Under-scaling and over-scaling are both suboptimal for all the obvious reasons. Middle-scaling is also suboptimal - you’re paying too much upfront and you don’t get enough later on anyway. Stage-appropriate scaling is the smart way. Scale as you can see the need. This requires willpower - you will be pressured to kick things down the road just one more time, every single time. The path of the wise is generational design. Build every component such that it can be easily destroyed and replaced. Anything that mustn’t be destroyed should be scrutinized and planned for in advance. Store your data in most mundane formats so it’s easier read. Keep interactions simple and need-to-know. These are platitudes, you will need to fill the planned obsolesce roadmap for yourself. This unites your hands make the quickest crappiest implementation at the moment secure in knowledge you can fix it or scale it as needed later.
macspoofing 6y ago
>People are always so quick to comment, “don’t worry about scaling... if you are successful, you can redesign at that point.” However, it isn’t always easy to redesign later. Well ... you're going to have to. You don't know what bottlenecks you're going to hit as you scale up, so why spend precious development time trying to guess which systems to optimize? And to be clear, 'cloud-scale' architectures are complicated and bring their own set of issues - why subject yourself to that? Plus a well designed (monolithic) system should get you very far with vertical (resource) scaling and/or data sharding.
wejick 6y ago
Like suggested by many people that the number of companies will not reach that scale, so starting small and simple is the most efficient. There are few things in mind that may help : 1. Define the domain segregation and be discipline on the implementation. It will not only help ease the process when the system need to be segregated into multiple service, but also help on organizing the team. 2. After segregating the domain, even if all of them live in the same database dont use `JOIN` as much as possible. Define the API for each domain and use it as the main communication. 3. As boring as it sounds, use boring technologies. Pick framework that has been used by many like Jango, Rails and Laravel. They offer not only tools and libraries but also programming pattern that will help us to implement domain segregation. The framework I mentioned have good established (way to work with if not integrated) ORM, event driven method, front end development and means to scale. 4. Usually postgreSQL is enough and can bring long way, just remember point 1 when designing the DB. I think those points are quite straightforward and doesn't make the development more difficult, while trying to be future proof.
joatmon-snoo 6y ago
There's horizontal scaling, and then there's horizontal scaling. A lot of software assumes a single server is fine and as a consequence doesn't even bother trying to abstract that away; the singleton nature of the software then percolates virally through it all. On the flip side, some software actually isolates their scaling limitations and pushes the bottleneck into a single part of the system. With the former, you're screwed if you hit your vertical scaling limits (to say nothing of what your system availability probably looks like, if scaling is actually a concern). With the latter, you can play games with your scaling, perhaps by shoving a hierarchical or replicated cache in front of your scaling bottleneck, and effectively buy time to build a solution with better scaling. (Or perhaps never bother with that, and just be acutely aware of your system's scaling dimensions, like StackOverflow.) There's no such thing as a system that truly scales infinitely, just systems whose scale limits exceed our imagination. (Admittedly, most bugs of the form "INT64_MAX overflow" are generally a leak and not the result of growth :).)
marcinzm 6y ago
Spending 5x the effort to redesign when you have 10x the money/resources isn't an unreasonable tradeoff. Since the vast majority of early stage startups fail spending money on things which don't matter until they succeed is not a good business decision. Of course, WeWork at this point had 1500 engineers so calling it a small startup or using it as a case study (either way) for small startups seems a bit silly.
namelosw 6y ago
People always assume either there would be high-velocity development or high-scalability architecture. In theory, this would be correct at the end of the day. But in fact, we're far from that equilibrium. For example, here we have these two major players: Ruby: High-velocity development; Decisions have been made so you don't need to make them; Suffer from scalability issues both performance-wise and architecturally in the endgame. Java: Much slower development; Could be fast if correctly architected, but doomed as well if done wrong; Require a hell of infrastructures to work well. But you can also find some superstars among the minor players without many trade-offs, for example: Elixir: Development velocity and decision-wise are very close to Ruby; Performance is on-par or faster than Java without decisions; Messaging and cache already baked-in the language and tech stack though you don't need them in the beginning; Hire Ruby programmers and just start rollin'
zozbot234 6y ago
This. The very fact that you might think your use case does not require Kafka is actually proof that your use case requires Kafka.

rntksi 6y ago

I have had the same issue, and wishing we thought about that up the road rather than down the road is exactly the feeling when we huddle up for team meetings on optimisation. We could not have any down time for most days of the year.

gfodor 6y ago

Everybody at a company experiencing scaling issues feels the way you do. The argument however is if the past went the way you wish it did, arguably the company you currently work for would not exist, because they would have lacked the knowledge that you have now to know which areas were the ones that would matter. Absent that knowledge, they would have built “scalable” systems that still had bottlenecks they could not yet see and also would have adopted extraneous technology that ultimately would be thrown out as the product discarded the parts that ended up not being critical to the value proposition uncovered. And of course, this whole process would have burned time, energy, and cash, not to mention shifted the nature of hires and the composition of the team itself, such that in sum it may have been a sufficient opportunity cost to jeopardize the whole business.

cortesoft 6y ago
Yeah, I know this is true, and we always try to remember to understand that the engineering decisions that were made at the time were made for good reasons given the circumstances we were facing. It helps that I was one of the people who made those decisions at the time, so I know we were making what we thought were the best decisions at the time. Still wish we had made some different ones, though!
- josephg 6y ago
  Hear hear! I worked at a startup a handful of years ago which decided early on that mongodb was good enough. We figured we'd move to a proper database down the road. Spoilers: That never happened - the longer we used mongo, the more expensive migrating became. We never did manage to justify the migration. As far as I know the team is still stuck with it, and paying the price. (If you're reading this and you know who you are, I'm sorry!)
  - rubyist5eva 6y ago
    Never use MongoDB. Ever. It's a rule of thumb that hasn't led me astray yet. http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/ http://www.sarahmei.com/blog/2013/11/11/why-you-should-never...
    - dominotw 6y ago
      > It's a rule of thumb that hasn't led me astray yet. How can you know if your decision is correct if you never use it in the first place. Isn't it like only implementing A version of A/B and concluding that not implementing B version was a good decision.
    - jcpst 6y ago
      Meh. I read the article, and I agree with it. But that is not what you are saying. Don’t use it for relational data. Don’t use it as your primary data store. At the enterprise I work at, I’ve had success delivering features where data science would have some job that aggregates production data and dumps it to kafka. Then we’ll map it and dump it in mongo. Stick a .net core microservice on top for retrieval. Works great, has none of the problems mentioned in that article. Of course, this might not make sense if you’re not big enough to where you need kafka, kubernetes, etc But “Never use it” is certainly not good advice.

louwrentius 6y ago

So, Let's Encrypt just uses a very big piece of iron for their database server, they don't scale horizontally. Stack Overflow also famously buys the biggest hardware they can get and upgrade their database server accordingly. I always wonder if horizontal scaling is really always the right answer, when you can just buy a bigger box (of bigger VM) and spare yourself a lot of issues.

basilgohar 6y ago
I am not a proponent of scaling out horizontally as the best solution to all things, but being able to do so comes as a result of design that helps to minimize coupling of services and single points of failure, in general, which are other benefits. But I also tend to think "you can always get a bigger box" can take you pretty far, and much further than most people that only think in terms of AWS realize.
cortesoft 6y ago
Up to a certain point, sure, you can vertically scale. I work for a large CDN, with many thousands of servers pushing near line-rate out of our NICs... we certainly can't vertically scale that. There is a limit to how much you can vertically scale. Even for the things we CAN vertically scale, we need to be able to handle hardware failure and OS upgrades without taking any downtime, so you are going to need to be able to horizontally scale enough to handle that.
- jorams 6y ago
  > I work for a large CDN, with many thousands of servers pushing near line-rate out of our NICs... we certainly can't vertically scale that. A CDN is essentially "horizontal scale as a service". The vast majority of products are very different.
  - hinkley 6y ago
    And the products that aren’t should hire out CDN networks to do that work for them instead of trying to roll their own.

ineedasername 6y ago

That's a bit of survivor bias: Most startups don't go from small to large, or even really medium. Most startups are strapped for resources. Paying an extra engineer, or taking valuable time away from an existing one to implement an extremely scalable infrastructure that might not be used. They should focus on shipping a good product that people want, and getting people to buy it. I can't think of startups that failed because their stack needed to be upgraded for scalability. It's generally not why startups fail. I can find a whole lot who burned through their cash before they had a chance to need something orders of magnitude larger in scale than currently necessary. If it's a minimal cost in time & money? Sure, go for high scalability. But it it's going to reduce your runway significantly then worry about the bigger problem of becoming successful enough to need it in the first place. If a startup is really concerned about avoiding the problems you're encountering at your company they should just take an extra week when planning their infrastructure to map out details of how they would transition to a more scalable infrastructure. Then save that file somewhere and dust it off if they ever need it. And if they need it, they'll have ample investment money coming their way to pay for the transition.

tomlagier 6y ago

My last 3 consulting gigs have all had the same general issue - they went with the "quick to start up", scalable solutions and immediately smacked into a brick wall of complexity. These are 1 to 3 person eng teams that started with serverless functions, NoSQL databases, and event-based architecture and not even 6 months down the line had to hire an expensive consultant to shake out their architecture and point them at something less scalable but more practical to build on at their size. If they didn't have VC, they'd likely die on the spot, drowned before they even had a shot at real product-market fit under the weight of their own accidental complexity. Now, is there a set of common patterns that works well enough for startups to iterate quickly with, that _also_ scales? Probably. Can you be successful with NoSQL et al out of the gate? Almost definitely. Does it make it _much harder_? I have three anecdotal data points that point to "yes". From my work over the last year, I'd suggest "choose boring technology" over "worry about scaling early" 10 out of 10 times. You need something that scales _conceptually_ before it can scale in the cloud, if you don't know what your core business value is or how your product actually works it's really damn hard to build a cloud-native app that anyone can actually iterate on. You're fighting incidental complexity every step of the way. Maybe in 5 or 10 years, when the average developer has a wealth of experience in scaling technologies, when serverless functions have fewer dramatic downsides, when we start seeing out-of-the-box event driven full stack frameworks, when the bleeding sharp edges are sanded off NoSQL and we've figured out a good story around schemas and validation, then it starts to look more attractive for young companies. Until then, though? Stick with the boring stuff. EDIT: Though, re-reading this, I suppose I should acknowledge my own survivorship bias - the companies that figured out their scalable, cloud-based tech stack early have no need for my consulting services...

sk5t 6y ago
"Choose Boring Technology": http://boringtechnology.club/ http://boringtechnology.club/
macspoofing 6y ago
Great comment. This mirrors the ridiculous 'big data' fad of a few years ago, where startups would deploy Hadoop clusters to crunch through Terabytes (or even Gigabytes!!) of data ... a task that could have been done much better by a well resourced monolithic system that was easy to implement, and easy to maintain and understand and just worked.

kortilla 6y ago

There is no startup in the scaling phase that doesn’t have stuff that needs to be rewritten. If your startup’s product is constantly iterating to find product market fit, it’s a fool’s errand to keep making sure everything is perfectly horizontally scalable in the first place. It’s very rare that the things you expect to be important early on actually are in the product that gets popular. You’re absolutely right to change your mantra now that you’re scaling, but you were right to not do it early on. The difference between life an death in an early startup is quick iteration.

dgb23 6y ago

I wonder what the ratio of failed startups that didn't engineer for scaling early vs those that did is. Having to rewrite parts of your application to match scaling issues seems like a great problem to have.

brightball 6y ago

There's two very different ways to interpret the "don't worry about scaling" stance and I think that's where things go sideways. On the one, people take it as literally "don't worry about it, just make it work" and don't think through simple decisions. On the other, with some basic analysis you can identify where your likely bottlenecks are going to be and make some preparations to dramatically simplify scaling when you need to do it. Sometimes minor architectural decisions, that don't take much time, can lead to a happier long term life. But I 100% agree with designing for horizontally scaling. It's one of the reasons I always try to steer people in the direction of Heroku for new projects, because while it reduces your ops burden and makes your life easier...it also forces you to design for horizontal scalability from day 1 via 12 Factor.

acdha 6y ago
That’s how I take this, with a related point of thinking about how you’d replace components when the time comes. Maybe you have a monolith-leaning design but you have tests, a clean API with some idea of ownership, etc. rather than a bunch of code directly making SQL calls, copying the same logic instead of using a function, or touching the shared files so when you do split things out and redesign them the process is safe. (Having flashbacks to the same code in 3 PHP repos, maintained by one person, no two copies identical)

Moto7451 6y ago

At my work we’re one step past where it sounds you’re at. I’ve been around since the very early days so I’m both part of the original problem and the solution. Now our biggest problem is we don’t like to work with the things we built to handle scale. I’m not sure how common that is but I don’t believe we’re alone. A lot of the problem from my point of view was trying to chase trends. We have Kafka in places the article’s author would suggest are inappropriate. We have a lot of “one true systems” that hid horizontally scaling instead of making it easy to build clusters of micro-services. Now we’re riding micro-services pretty hard and that isn’t going well since we have too many mixed architectures in use. I think it’s correct to push out worrying about scale while you seek product market fit. I think that once you do need to scale you really need to put thought into the process. This is difficult if everything is on fire and failing. If it’s not, burn some time to plan. It’ll pay off. Be sure you don’t reinvent things you can cheaply buy. Make sure developers still like their day jobs after the changes are made before making more changes with the same underlying design theories.

Person5478 6y ago
As companies grow there will be "pain points" that come up due to scaling (performance, architecture, development, etc). When you get to that point it's normal to wish the pain didn't exist by "doing it right" the first time, but I think it's still better to wait until you have the pain. Because that pain tells you unequivocally what you need to value in the scaling.

bostonsre 6y ago

Yea, I don't think it's always a waste of time to think about scale at the start. You don't have to do all the work up front, but you can definitely put thought into doing the early work in a way that will make scaling easier later. There is no sense in committing to an architecture early that limits your options later. Also, avoiding sloppiness and ensuring simplicity up front usually pays dividends later.

DenisM 6y ago

Under-scaling and over-scaling are both suboptimal for all the obvious reasons. Middle-scaling is also suboptimal - you’re paying too much upfront and you don’t get enough later on anyway. Stage-appropriate scaling is the smart way. Scale as you can see the need. This requires willpower - you will be pressured to kick things down the road just one more time, every single time. The path of the wise is generational design. Build every component such that it can be easily destroyed and replaced. Anything that mustn’t be destroyed should be scrutinized and planned for in advance. Store your data in most mundane formats so it’s easier read. Keep interactions simple and need-to-know. These are platitudes, you will need to fill the planned obsolesce roadmap for yourself. This unites your hands make the quickest crappiest implementation at the moment secure in knowledge you can fix it or scale it as needed later.

macspoofing 6y ago

>People are always so quick to comment, “don’t worry about scaling... if you are successful, you can redesign at that point.” However, it isn’t always easy to redesign later. Well ... you're going to have to. You don't know what bottlenecks you're going to hit as you scale up, so why spend precious development time trying to guess which systems to optimize? And to be clear, 'cloud-scale' architectures are complicated and bring their own set of issues - why subject yourself to that? Plus a well designed (monolithic) system should get you very far with vertical (resource) scaling and/or data sharding.

wejick 6y ago

Like suggested by many people that the number of companies will not reach that scale, so starting small and simple is the most efficient. There are few things in mind that may help : 1. Define the domain segregation and be discipline on the implementation. It will not only help ease the process when the system need to be segregated into multiple service, but also help on organizing the team. 2. After segregating the domain, even if all of them live in the same database dont use `JOIN` as much as possible. Define the API for each domain and use it as the main communication. 3. As boring as it sounds, use boring technologies. Pick framework that has been used by many like Jango, Rails and Laravel. They offer not only tools and libraries but also programming pattern that will help us to implement domain segregation. The framework I mentioned have good established (way to work with if not integrated) ORM, event driven method, front end development and means to scale. 4. Usually postgreSQL is enough and can bring long way, just remember point 1 when designing the DB. I think those points are quite straightforward and doesn't make the development more difficult, while trying to be future proof.

joatmon-snoo 6y ago

There's horizontal scaling, and then there's horizontal scaling. A lot of software assumes a single server is fine and as a consequence doesn't even bother trying to abstract that away; the singleton nature of the software then percolates virally through it all. On the flip side, some software actually isolates their scaling limitations and pushes the bottleneck into a single part of the system. With the former, you're screwed if you hit your vertical scaling limits (to say nothing of what your system availability probably looks like, if scaling is actually a concern). With the latter, you can play games with your scaling, perhaps by shoving a hierarchical or replicated cache in front of your scaling bottleneck, and effectively buy time to build a solution with better scaling. (Or perhaps never bother with that, and just be acutely aware of your system's scaling dimensions, like StackOverflow.) There's no such thing as a system that truly scales infinitely, just systems whose scale limits exceed our imagination. (Admittedly, most bugs of the form "INT64_MAX overflow" are generally a leak and not the result of growth :).)

marcinzm 6y ago

Spending 5x the effort to redesign when you have 10x the money/resources isn't an unreasonable tradeoff. Since the vast majority of early stage startups fail spending money on things which don't matter until they succeed is not a good business decision. Of course, WeWork at this point had 1500 engineers so calling it a small startup or using it as a case study (either way) for small startups seems a bit silly.

namelosw 6y ago

People always assume either there would be high-velocity development or high-scalability architecture. In theory, this would be correct at the end of the day. But in fact, we're far from that equilibrium. For example, here we have these two major players: Ruby: High-velocity development; Decisions have been made so you don't need to make them; Suffer from scalability issues both performance-wise and architecturally in the endgame. Java: Much slower development; Could be fast if correctly architected, but doomed as well if done wrong; Require a hell of infrastructures to work well. But you can also find some superstars among the minor players without many trade-offs, for example: Elixir: Development velocity and decision-wise are very close to Ruby; Performance is on-par or faster than Java without decisions; Messaging and cache already baked-in the language and tech stack though you don't need them in the beginning; Hire Ruby programmers and just start rollin'

zozbot234 6y ago

This. The very fact that you might think your use case does not require Kafka is actually proof that your use case requires Kafka.