11 ms·

As someone bootstrapping a business, my cost structure favors horizontally scaling databases with low/no fixed overhead vs vertically scaling databases with hig

As someone bootstrapping a business, my cost structure favors horizontally scaling databases with low/no fixed overhead vs vertically scaling databases with high overhead. And I don’t think this is exactly an uncommon situation - plenty of people, even within large well-funded organizations, would have a hard time justifying such a large expenditure if it were uncertain that it’d be fully utilized. It’s also a lot easier for me to handle tenancy/isolation/different data “realms” this way. And I sleep better at night knowing a mistake (I have no dbas, and I don’t have time to invest in tons of safeguards as a solo dev when there are much higher EV tasks) won’t nuke absolutely everything.

smt88 3y ago
For small companies, I've never started with horizontal scaling. With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that generates suboptimal queries. If I have horizontal scaling, sometimes even a reasonable query that's missing an explicit index can turn into a 30s query that I have to spend dev time debugging.
- kstrauser 3y ago
  I wish I could upvote this more. 95+% of startups will never need to scale past one reasonably sized single-instance database. Maybe they'll want to have a couple of replicas running for a fast failover, but from a performance perspective, they should spend their time writing efficient queries instead of horizontally scaling. It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.
  - pi-e-sigma 3y ago
    I think most even semi-experienced developers know this. But doing the right thing is against their career grow goals so they overcomplicate the architecture on purpose.
    - collyw 3y ago
      This industry is way too much of a fashion show. And too many devs are consumers of new technology in a similar way. After 20 years I am actually getting a bit skeptical behind the "you should constantly be learning" mantra. If you are constantly learning new, tech, you will never be an expert in anything. (That statement is very much a generalization and I am sure there are tons of people that will point out issues with it).
      - justincredible 3y ago
        [dead]
    - The_Colonel 3y ago
      > I think most even semi-experienced developers know this. I've met plenty of pretty senior people believing the opposite and most of them likely aren't good liars.
    - wharvle 3y ago
      You also often have to fight for the right thing, then any problems that do happen are your fault. When the dumbshit over engineered "cloud architecture" that was pushed on you (managers don't get promoted or switch companies for a big raise by reducing their headcount... no, they get promoted by building a big team for a successful [please don't check in on how it was doing two years later...] "cloud transformation") has all kinds of problems and the costs shoot into the stratosphere, while five servers on a rack would have granted greater uptime and performance in practice for your usage patterns, and lower costs... it's not your fault. Have some bad luck and that simple solution you pushed for happens to experience an unlikely failure causing a significant outage in year 1, and your head's on the chopping block.
      - pi-e-sigma 3y ago
        What I experienced is that it doesn't matter if I pushed for the right thing or inherited some overcomplicated shit. In case of any problems it's always my fault anyway. You just can't win.
- opportune 3y ago
  To elaborate, I’m using managed databases that abstract away scaling so that my UX as a developer is that they are scaling horizontally (in data, and cost). While I’m pre-launch and doing testing, using a more traditional vertical model is money out of my pocket. And it’s not something I want to maintain after I launch either. Also, I’m creating something (a freemium game) where as a solo dev it will only be worth it for me to continue working on the business if it sees explosive growth. If I’m lucky enough for that to materialize, there will be at least a few months before I can hire fulltime help for the product (while also being busy doing tons of other stuff) - it would be terrible to have to handle tons of db operations tasks or re-architect things to support growth during that time. Basically, vertical scaling is optimizing for the wrong level of usage. It’s actually the same for many of the snarky “your startup doesn’t need horizontal scaling” comments here - if your states are {building, huge growth, failure} then horizontal scaling is perfect because it keeps your costs low while you’re building and handles the huge growth phase more gracefully. Yes, maybe you’ll never get to the huge growth phase, but the entire business is oriented around either getting there or failing.
  - kstrauser 3y ago
    I certainly didn't mean my comment to be snarky. However, I do want to be realistic. There's an exception for every guideline. Maybe yours is that exception. I worked at a company like you described, where we had a few tens of thousands of users, but were actively in talks with a partner that would have given us international press, and who would have taken us to about 20 million concurrent users a month later. (Imagine a beer company telling their customers to use our app to interact with the Super Bowl broadcast. That sort of thing.) We spent a lot of time talking about horizontal scaling because that was the whole point of the company. Maybe yours is like that, too. But the vast majority of small companies are not like that. They acquire one customer, then another, and another, and grow N<=50% per year, with tons of headroom on default app configurations. For those companies, worrying about explosive, global scaling instead of, you know, building their business, is time they can't get back.
    - collyw 3y ago
      I worked for a company. The system was basically a crud web app, where users could upload medical imaging data, and send it to be processed in the background. We had the web app set up on Kubernetes, yet the background jobs had to be manually managed by admins at the company. There were less than 300 users in total and the only time Kubernetes scaled beyond one pod was when something was broken. When I joined, they had just implemented the Kubernetes stuff. I thought it was kind of overkill, but as the new guy I didn't feel it was my place to point it out. Buy hey, some people got some nice experience on their resumes.
__s 3y ago
One database as source of truth, & backups that are tested. Use replication to run analysis on Citus offered being able to have most of your database be plain postgres & then having the few tables that take up most the database be distributed But for a small company you probably don't need scaling period
kaba0 3y ago
Do you ever get to a place where you actually have to scale it? Like, the PC from my teenager age would probably be more than fast enough for 80% of companies' data. Also, are you sure your data is actually in a correct view across all these "realms"?
- opportune 3y ago
  Basically I’m optimizing for viral growth, otherwise the business probably isn’t worth it. I haven’t launched yet but I estimate that at 100k DAU vertical scaling/using a single instance would become a nightmare because of throughput and latency rather than data size. I’m admittedly using a strange architecture for my use case and I realize now commenting here opened too big a can of worms as explaining exactly what I’m doing would derail the thread. Suffice it to say, my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data. But yes, using a more horizontal database strategy makes it very easy to manage data across realms. That’s one of the main benefits. A single DB would be much harder as far as isolation and separating test/production traffic (assuming this is what you mean by views) than having multiple separable dbs that only talk to dev/staging. And I can easily wipe out and isolate dev and staging this way. I’m frankly shocked people would advocate a single db that doesn’t allow you to do this.
  - tomnipotent 3y ago
    > that at 100k DAU vertical That's chump change size even for a medium EC2/RDS instance, which should be capable of tens of millions of queries a day without the CPU or disk complaining at you (unless all your queries are table scans or unindexed). > my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data It doesn't matter, it's still just bytes. What will matter is your query pattern relative to the databases query planner efficacy, and how updates/deletes impact this. > makes it very easy to manage data across realms You can just as easily do this at first as separate databases/schemas on the same physical server, with different users and permissions to prevent cross-database/schema joins so that when you need to move them to different machines it's an easier process. Everyone I know that has tested isolated multi-tenancy that wasn't dependent on legal needs ended up abandoning this approach and consolidating into as little hardware as possible. Heap Analytics had a blog post a few years ago about this, but I can't seem to find it. Regardless, hope you success in your endeavor and that you come back in a few months to prove us all wrong.

smt88 3y ago

For small companies, I've never started with horizontal scaling. With vertical scaling and maybe 2-3 instances, I don't have to worry about hasty ORM code that generates suboptimal queries. If I have horizontal scaling, sometimes even a reasonable query that's missing an explicit index can turn into a 30s query that I have to spend dev time debugging.

kstrauser 3y ago
I wish I could upvote this more. 95+% of startups will never need to scale past one reasonably sized single-instance database. Maybe they'll want to have a couple of replicas running for a fast failover, but from a performance perspective, they should spend their time writing efficient queries instead of horizontally scaling. It's good to think about the what-ifs along the way. Can we shard this setup if we land a huge customer, or one that's hyper-sensitive about having their data separated? If/when that time comes, you'll have ideas of what to do about it. But realistically, most companies never hit that point. Every hour spent worrying how to make their product FAANG-scale is an hour spent not making their product.
- pi-e-sigma 3y ago
  I think most even semi-experienced developers know this. But doing the right thing is against their career grow goals so they overcomplicate the architecture on purpose.
  - collyw 3y ago
    This industry is way too much of a fashion show. And too many devs are consumers of new technology in a similar way. After 20 years I am actually getting a bit skeptical behind the "you should constantly be learning" mantra. If you are constantly learning new, tech, you will never be an expert in anything. (That statement is very much a generalization and I am sure there are tons of people that will point out issues with it).
    - justincredible 3y ago
      [dead]
  - The_Colonel 3y ago
    > I think most even semi-experienced developers know this. I've met plenty of pretty senior people believing the opposite and most of them likely aren't good liars.
  - wharvle 3y ago
    You also often have to fight for the right thing, then any problems that do happen are your fault. When the dumbshit over engineered "cloud architecture" that was pushed on you (managers don't get promoted or switch companies for a big raise by reducing their headcount... no, they get promoted by building a big team for a successful [please don't check in on how it was doing two years later...] "cloud transformation") has all kinds of problems and the costs shoot into the stratosphere, while five servers on a rack would have granted greater uptime and performance in practice for your usage patterns, and lower costs... it's not your fault. Have some bad luck and that simple solution you pushed for happens to experience an unlikely failure causing a significant outage in year 1, and your head's on the chopping block.
    - pi-e-sigma 3y ago
      What I experienced is that it doesn't matter if I pushed for the right thing or inherited some overcomplicated shit. In case of any problems it's always my fault anyway. You just can't win.
opportune 3y ago
To elaborate, I’m using managed databases that abstract away scaling so that my UX as a developer is that they are scaling horizontally (in data, and cost). While I’m pre-launch and doing testing, using a more traditional vertical model is money out of my pocket. And it’s not something I want to maintain after I launch either. Also, I’m creating something (a freemium game) where as a solo dev it will only be worth it for me to continue working on the business if it sees explosive growth. If I’m lucky enough for that to materialize, there will be at least a few months before I can hire fulltime help for the product (while also being busy doing tons of other stuff) - it would be terrible to have to handle tons of db operations tasks or re-architect things to support growth during that time. Basically, vertical scaling is optimizing for the wrong level of usage. It’s actually the same for many of the snarky “your startup doesn’t need horizontal scaling” comments here - if your states are {building, huge growth, failure} then horizontal scaling is perfect because it keeps your costs low while you’re building and handles the huge growth phase more gracefully. Yes, maybe you’ll never get to the huge growth phase, but the entire business is oriented around either getting there or failing.
- kstrauser 3y ago
  I certainly didn't mean my comment to be snarky. However, I do want to be realistic. There's an exception for every guideline. Maybe yours is that exception. I worked at a company like you described, where we had a few tens of thousands of users, but were actively in talks with a partner that would have given us international press, and who would have taken us to about 20 million concurrent users a month later. (Imagine a beer company telling their customers to use our app to interact with the Super Bowl broadcast. That sort of thing.) We spent a lot of time talking about horizontal scaling because that was the whole point of the company. Maybe yours is like that, too. But the vast majority of small companies are not like that. They acquire one customer, then another, and another, and grow N<=50% per year, with tons of headroom on default app configurations. For those companies, worrying about explosive, global scaling instead of, you know, building their business, is time they can't get back.
  - collyw 3y ago
    I worked for a company. The system was basically a crud web app, where users could upload medical imaging data, and send it to be processed in the background. We had the web app set up on Kubernetes, yet the background jobs had to be manually managed by admins at the company. There were less than 300 users in total and the only time Kubernetes scaled beyond one pod was when something was broken. When I joined, they had just implemented the Kubernetes stuff. I thought it was kind of overkill, but as the new guy I didn't feel it was my place to point it out. Buy hey, some people got some nice experience on their resumes.

__s 3y ago

One database as source of truth, & backups that are tested. Use replication to run analysis on Citus offered being able to have most of your database be plain postgres & then having the few tables that take up most the database be distributed But for a small company you probably don't need scaling period

kaba0 3y ago

Do you ever get to a place where you actually have to scale it? Like, the PC from my teenager age would probably be more than fast enough for 80% of companies' data. Also, are you sure your data is actually in a correct view across all these "realms"?

opportune 3y ago
Basically I’m optimizing for viral growth, otherwise the business probably isn’t worth it. I haven’t launched yet but I estimate that at 100k DAU vertical scaling/using a single instance would become a nightmare because of throughput and latency rather than data size. I’m admittedly using a strange architecture for my use case and I realize now commenting here opened too big a can of worms as explaining exactly what I’m doing would derail the thread. Suffice it to say, my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data. But yes, using a more horizontal database strategy makes it very easy to manage data across realms. That’s one of the main benefits. A single DB would be much harder as far as isolation and separating test/production traffic (assuming this is what you mean by views) than having multiple separable dbs that only talk to dev/staging. And I can easily wipe out and isolate dev and staging this way. I’m frankly shocked people would advocate a single db that doesn’t allow you to do this.
- tomnipotent 3y ago
  > that at 100k DAU vertical That's chump change size even for a medium EC2/RDS instance, which should be capable of tens of millions of queries a day without the CPU or disk complaining at you (unless all your queries are table scans or unindexed). > my db doesn’t contain just “Bob ordered 5 widgets to Foo Lane” data It doesn't matter, it's still just bytes. What will matter is your query pattern relative to the databases query planner efficacy, and how updates/deletes impact this. > makes it very easy to manage data across realms You can just as easily do this at first as separate databases/schemas on the same physical server, with different users and permissions to prevent cross-database/schema joins so that when you need to move them to different machines it's an easier process. Everyone I know that has tested isolated multi-tenancy that wasn't dependent on legal needs ended up abandoning this approach and consolidating into as little hardware as possible. Heap Analytics had a blog post a few years ago about this, but I can't seem to find it. Regardless, hope you success in your endeavor and that you come back in a few months to prove us all wrong.