5 ms·

For small companies, I've never started with horizontal scaling. With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that

by smt88 3y ago

For small companies, I've never started with horizontal scaling. With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that generates suboptimal queries. If I have horizontal scaling, sometimes even a reasonable query that's missing an explicit index can turn into a 30s query that I have to spend dev time debugging.

kstrauser 3y ago
I wish I could upvote this more. 95+% of startups will never need to scale past one reasonably sized single-instance database. Maybe they'll want to have a couple of replicas running for a fast failover, but from a performance perspective, they should spend their time writing efficient queries instead of horizontally scaling. It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.
- pi-e-sigma 3y ago
  I think most even semi-experienced developers know this. But doing the right thing is against their career grow goals so they overcomplicate the architecture on purpose.
  - collyw 3y ago
    This industry is way too much of a fashion show. And too many devs are consumers of new technology in a similar way. After 20 years I am actually getting a bit skeptical behind the "you should constantly be learning" mantra. If you are constantly learning new, tech, you will never be an expert in anything. (That statement is very much a generalization and I am sure there are tons of people that will point out issues with it).
    - justincredible 3y ago
      [dead]
  - The_Colonel 3y ago
    > I think most even semi-experienced developers know this. I've met plenty of pretty senior people believing the opposite and most of them likely aren't good liars.
  - wharvle 3y ago
    You also often have to fight for the right thing, then any problems that do happen are your fault. When the dumbshit over engineered "cloud architecture" that was pushed on you (managers don't get promoted or switch companies for a big raise by reducing their headcount... no, they get promoted by building a big team for a successful [please don't check in on how it was doing two years later...] "cloud transformation") has all kinds of problems and the costs shoot into the stratosphere, while five servers on a rack would have granted greater uptime and performance in practice for your usage patterns, and lower costs... it's not your fault. Have some bad luck and that simple solution you pushed for happens to experience an unlikely failure causing a significant outage in year 1, and your head's on the chopping block.
    - pi-e-sigma 3y ago
      What I experienced is that it doesn't matter if I pushed for the right thing or inherited some overcomplicated shit. In case of any problems it's always my fault anyway. You just can't win.
      - bob1029 3y ago
        If you want the latitude to affect meaningful change, then you are also going to have to accept all ownership surrounding that thing and likely far beyond. I know it gets some flak (for good reason), but concepts like extreme ownership are fundamentally how I am able to tolerate this "everything is my fault" experience. If everything will be my fault in the end, then screw it - I will go ahead and take ownership/control of all of those things and make sure they don't come back around to bite me in the ass. If someone questions my authority, I always cast it in terms of "how many weekends with your family are you willing to sacrifice to support that idea?" Oftentimes this is all it takes to get those idle suggestions put in the bin so we can move forward. Solving database circus in a real business environment requires disengaging all of the safeties and getting very messy with your free time, stress levels and other humans. It's a front-loaded suffering though. If you actually know what you are doing and get through the schema refactor (and it sticks), everything is down-hill from that point. Getting one schema to rule them all and then having the team learn that schema == new common language by which you can communicate that did not exist prior. In our business, non-developers are just as aware of our tables, columns and relations as developers & DBAs are. In some cases, they have more detail about how a field is used than I do.
  - fullstackchris 3y ago
    > doing the right thing is against their career growth goals only if the people who are hiring you are just as ignorant
opportune 3y ago
To elaborate, I’m using managed databases that abstract away scaling so that my UX as a developer is that they are scaling horizontally (in data, and cost). While I’m pre-launch and doing testing, using a more traditional vertical model is money out of my pocket. And it’s not something I want to maintain after I launch either. Also, I’m creating something (a freemium game) where as a solo dev it will only be worth it for me to continue working on the business if it sees explosive growth. If I’m lucky enough for that to materialize, there will be at least a few months before I can hire fulltime help for the product (while also being busy doing tons of other stuff) - it would be terrible to have to handle tons of db operations tasks or re-architect things to support growth during that time. Basically, vertical scaling is optimizing for the wrong level of usage. It’s actually the same for many of the snarky “your startup doesn’t need horizontal scaling” comments here - if your states are {building, huge growth, failure} then horizontal scaling is perfect because it keeps your costs low while you’re building and handles the huge growth phase more gracefully. Yes, maybe you’ll never get to the huge growth phase, but the entire business is oriented around either getting there or failing.
- kstrauser 3y ago
  I certainly didn't mean my comment to be snarky. However, I do want to be realistic. There's an exception for every guideline. Maybe yours is that exception. I worked at a company like you described, where we had a few tens of thousands of users, but were actively in talks with a partner that would have given us international press, and who would have taken us to about 20 million concurrent users a month later. (Imagine a beer company telling their customers to use our app to interact with the Super Bowl broadcast. That sort of thing.) We spent a lot of time talking about horizontal scaling because that was the whole point of the company. Maybe yours is like that, too. But the vast majority of small companies are not like that. They acquire one customer, then another, and another, and grow N<=50% per year, with tons of headroom on default app configurations. For those companies, worrying about explosive, global scaling instead of, you know, building their business, is time they can't get back.
  - collyw 3y ago
    I worked for a company. The system was basically a crud web app, where users could upload medical imaging data, and send it to be processed in the background. We had the web app set up on Kubernetes, yet the background jobs had to be manually managed by admins at the company. There were less than 300 users in total and the only time Kubernetes scaled beyond one pod was when something was broken. When I joined, they had just implemented the Kubernetes stuff. I thought it was kind of overkill, but as the new guy I didn't feel it was my place to point it out. Buy hey, some people got some nice experience on their resumes.

by smt88 3y ago

kstrauser 3y ago

I wish I could upvote this more. 95+% of startups will never need to scale past one reasonably sized single-instance database. Maybe they'll want to have a couple of replicas running for a fast failover, but from a performance perspective, they should spend their time writing efficient queries instead of horizontally scaling. It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.

pi-e-sigma 3y ago
I think most even semi-experienced developers know this. But doing the right thing is against their career grow goals so they overcomplicate the architecture on purpose.
- collyw 3y ago
  This industry is way too much of a fashion show. And too many devs are consumers of new technology in a similar way. After 20 years I am actually getting a bit skeptical behind the "you should constantly be learning" mantra. If you are constantly learning new, tech, you will never be an expert in anything. (That statement is very much a generalization and I am sure there are tons of people that will point out issues with it).
  - justincredible 3y ago
    [dead]
- The_Colonel 3y ago
  > I think most even semi-experienced developers know this. I've met plenty of pretty senior people believing the opposite and most of them likely aren't good liars.
- wharvle 3y ago
  You also often have to fight for the right thing, then any problems that do happen are your fault. When the dumbshit over engineered "cloud architecture" that was pushed on you (managers don't get promoted or switch companies for a big raise by reducing their headcount... no, they get promoted by building a big team for a successful [please don't check in on how it was doing two years later...] "cloud transformation") has all kinds of problems and the costs shoot into the stratosphere, while five servers on a rack would have granted greater uptime and performance in practice for your usage patterns, and lower costs... it's not your fault. Have some bad luck and that simple solution you pushed for happens to experience an unlikely failure causing a significant outage in year 1, and your head's on the chopping block.
  - pi-e-sigma 3y ago
    What I experienced is that it doesn't matter if I pushed for the right thing or inherited some overcomplicated shit. In case of any problems it's always my fault anyway. You just can't win.
    - bob1029 3y ago
      If you want the latitude to affect meaningful change, then you are also going to have to accept all ownership surrounding that thing and likely far beyond. I know it gets some flak (for good reason), but concepts like extreme ownership are fundamentally how I am able to tolerate this "everything is my fault" experience. If everything will be my fault in the end, then screw it - I will go ahead and take ownership/control of all of those things and make sure they don't come back around to bite me in the ass. If someone questions my authority, I always cast it in terms of "how many weekends with your family are you willing to sacrifice to support that idea?" Oftentimes this is all it takes to get those idle suggestions put in the bin so we can move forward. Solving database circus in a real business environment requires disengaging all of the safeties and getting very messy with your free time, stress levels and other humans. It's a front-loaded suffering though. If you actually know what you are doing and get through the schema refactor (and it sticks), everything is down-hill from that point. Getting one schema to rule them all and then having the team learn that schema == new common language by which you can communicate that did not exist prior. In our business, non-developers are just as aware of our tables, columns and relations as developers & DBAs are. In some cases, they have more detail about how a field is used than I do.
- fullstackchris 3y ago
  > doing the right thing is against their career growth goals only if the people who are hiring you are just as ignorant

opportune 3y ago

To elaborate, I’m using managed databases that abstract away scaling so that my UX as a developer is that they are scaling horizontally (in data, and cost). While I’m pre-launch and doing testing, using a more traditional vertical model is money out of my pocket. And it’s not something I want to maintain after I launch either. Also, I’m creating something (a freemium game) where as a solo dev it will only be worth it for me to continue working on the business if it sees explosive growth. If I’m lucky enough for that to materialize, there will be at least a few months before I can hire fulltime help for the product (while also being busy doing tons of other stuff) - it would be terrible to have to handle tons of db operations tasks or re-architect things to support growth during that time. Basically, vertical scaling is optimizing for the wrong level of usage. It’s actually the same for many of the snarky “your startup doesn’t need horizontal scaling” comments here - if your states are {building, huge growth, failure} then horizontal scaling is perfect because it keeps your costs low while you’re building and handles the huge growth phase more gracefully. Yes, maybe you’ll never get to the huge growth phase, but the entire business is oriented around either getting there or failing.

kstrauser 3y ago
I certainly didn't mean my comment to be snarky. However, I do want to be realistic. There's an exception for every guideline. Maybe yours is that exception. I worked at a company like you described, where we had a few tens of thousands of users, but were actively in talks with a partner that would have given us international press, and who would have taken us to about 20 million concurrent users a month later. (Imagine a beer company telling their customers to use our app to interact with the Super Bowl broadcast. That sort of thing.) We spent a lot of time talking about horizontal scaling because that was the whole point of the company. Maybe yours is like that, too. But the vast majority of small companies are not like that. They acquire one customer, then another, and another, and grow N<=50% per year, with tons of headroom on default app configurations. For those companies, worrying about explosive, global scaling instead of, you know, building their business, is time they can't get back.
- collyw 3y ago
  I worked for a company. The system was basically a crud web app, where users could upload medical imaging data, and send it to be processed in the background. We had the web app set up on Kubernetes, yet the background jobs had to be manually managed by admins at the company. There were less than 300 users in total and the only time Kubernetes scaled beyond one pod was when something was broken. When I joined, they had just implemented the Kubernetes stuff. I thought it was kind of overkill, but as the new guy I didn't feel it was my place to point it out. Buy hey, some people got some nice experience on their resumes.